Hi David,
Currently there is no way to write headers to HDFS using the built-in
Flume functionality.
If you are writing to text or binary files on HDFS (i.e. you have set
hdfs.fileType = DataStream or CompressedStream in your config), then you
can supply your own custom serializer, which will allow you to write
headers to HDFS. You will need to write a serializer that implements
org.apache.flume.serialization.EventSerializer.
If, on the other hand, you are writing to HDFS SequenceFiles, then
unfortunately there is no way to customize the way that events are
serialized, so you cannot write event headers to HDFS. This is a known
issue (FLUME-1100) and I have supplied a patch to fix it.
Chris.
On 2012/08/21 11:36, David Capwell wrote:
I was wondering if I pass random data to an event's header, can the
HDFSSink write it to HDFS? I know it can use the headers to split the
data into different paths, but what about writing the data to HDFS
itself?
thanks for your time reading this email.
