Reasons are mostly historical. Feel free to submit a patch to bump it up
Thanks, Hari On Thu, Dec 4, 2014 at 12:55 AM, Jean-Philippe Caruana <[email protected]> wrote: > Yes, flume-env.sh starts the JVM with 20 Mo and GCS opens a 64 Mo buffer. > Any idea/reason why flume starts with such a low heap space ? > Le 04/12/2014 01:19, Hari Shreedharan a écrit : >> It looks like you are just running out of heap space. Try increasing >> the heap space by specifying a higher value in the flume-env.sh file. >> >> Thanks, >> Hari >> >> >> On Mon, Dec 1, 2014 at 9:02 AM, Jean-Philippe Caruana >> <[email protected] <mailto:[email protected]>> wrote: >> >> I also asked the question on SO : >> >> https://stackoverflow.com/questions/27232966/what-causes-flume-with-gcs-sink-to-throw-a-outofmemoryexepction >> >> >> Le 01/12/2014 15:35, Jean-Philippe Caruana a écrit : >>> Hi, >>> >>> I managed to write to GS from flume [1], but this is not working >>> 100% yet: >>> - files are created in the expected directories, but are empty >>> - flume throws a java.lang.OutOfMemoryError: Java heap space: >>> >>> java.lang.OutOfMemoryError: Java heap space >>> at >>> java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76) >>> at >>> >>> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.<init>(GoogleHadoopOutputStream.java:79) >>> at >>> >>> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:820) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773) >>> at >>> >>> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:96) >>> >>> (complete stack trace here: http://pastebin.com/i5iSgCM3) >>> >>> Has anyone already experienced this ? >>> Is it a bug from google's gcs-connector-latest-hadoop2.jar ? >>> Where should I look to find out what's wrong ? >>> >>> My configuration looks like this: >>> a1.sinks.hdfs_sink.hdfs.path = >>> gs://bucket_name/%{env}/%{tenant}/%{type}/%Y-%m-%d >>> >>> I am running flume from Docker. >>> >>> [1] >>> >>> http://stackoverflow.com/questions/27174033/what-is-the-minimal-setup-needed-to-write-to-hdfs-gs-on-google-cloud-storage-wit >>> >>> Thanks. >> >> -- >> Jean-Philippe Caruana >> http://www.barreverte.fr >> >> > -- > Jean-Philippe Caruana - [email protected] > Target2sell, le turbo du e-commerce > 43 rue de Turbigo - 75003 Paris > +33 (0) 9 51 92 63 20 | +33 (0) 1 44 54 94 55 > http://www.target2sell.com > http://www.barreverte.fr
