I have the following HDFS Sink configuration which rolls files based on size. I am not able to get flume to close the last temp file before it moves to the next directory.
Do the configuration options below seem right? agent.sinks.HDFSSink.hdfs.rollInterval = 0 agent.sinks.HDFSSink.hdfs.rollSize = 512000000 agent.sinks.HDFSSink.hdfs.rollCount = 0 agent.sinks.HDFSSink.hdfs.batchSize = 10000 agent.sinks.HDFSSink.hdfs.fileType = CompressedStream agent.sinks.HDFSSink.hdfs.codeC = snappy agent.sinks.HDFSSink.hdfs.maxOpenFiles = 50 agent.sinks.HDFSSink.hdfs.appendTimeout = 10000 agent.sinks.HDFSSink.hdfs.callTimeout = 100 agent.sinks.HDFSSink.hdfs.threadsPoolSize = 100 agent.sinks.HDFSSink.hdfs.rollTimerPoolSize = 1Listing the files in HDFS for two directories look like the following: [majid@srv01 ~]$ hadoop fs -ls /user/monitor/incoming/2015/02/22/am/ | tail -5 -rw-r--r-- 3 flume flume 129204066 2015-02-22 11:24 /user/monitor/incoming/2015/02/22/am/FlumeData.1424563206488.snappy -rw-r--r-- 3 flume flume 129129935 2015-02-22 11:33 /user/monitor/incoming/2015/02/22/am/FlumeData.1424563206489.snappy -rw-r--r-- 3 flume flume 129224836 2015-02-22 11:43 /user/monitor/incoming/2015/02/22/am/FlumeData.1424563206490.snappy -rw-r--r-- 3 flume flume 130160914 2015-02-22 11:54 /user/monitor/incoming/2015/02/22/am/FlumeData.1424563206491.snappy -rw-r--r-- 3 flume flume 5123 2015-02-22 11:54 /user/monitor/incoming/2015/02/22/am/FlumeData.1424563206492.snappy.tmp [majid@srv01 ~]$ hadoop fs -ls /user/monitor/incoming/2015/02/22/pm/ | tail -5 -rw-r--r-- 3 flume flume 128659488 2015-02-22 23:19 /user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408953.snappy -rw-r--r-- 3 flume flume 127512784 2015-02-22 23:30 /user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408954.snappy -rw-r--r-- 3 flume flume 128234258 2015-02-22 23:41 /user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408955.snappy -rw-r--r-- 3 flume flume 128191069 2015-02-22 23:53 /user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408956.snappy -rw-r--r-- 3 flume flume 818575 2015-02-22 23:53 /user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408957.snappy.tmp Thanks, Majid
