Hi,
I am working on a poc to write to hdfs files using BucketingSink class. Even
thought I am the data is being writing to hdfs files, but the files are lying
with “.pending” on hdfs.
Below is the code I am using. Can someone pls help me identify the issue and
help me fix this ?
BucketingSink<String> HdfsSink = new
BucketingSink<String>("hdfs://xxxx/xxxx/xxxx/Test/");
HdfsSink.setBucketer(new DateTimeBucketer<String>("yyyy-MM-dd--HHmm"));
HdfsSink.setBatchSize(1024 * 1024 * 2); // this is 2 MB,
HdfsSink.setInactiveBucketCheckInterval(10000L);
HdfsSink.setInactiveBucketThreshold(10000L);
Thanks a lot.
Regards,
Raja.
