Hi, I am working on a poc to write to hdfs files using BucketingSink class. Even thought I am the data is being writing to hdfs files, but the files are lying with “.pending” on hdfs.
Below is the code I am using. Can someone pls help me identify the issue and help me fix this ? BucketingSink<String> HdfsSink = new BucketingSink<String>("hdfs://xxxx/xxxx/xxxx/Test/"); HdfsSink.setBucketer(new DateTimeBucketer<String>("yyyy-MM-dd--HHmm")); HdfsSink.setBatchSize(1024 * 1024 * 2); // this is 2 MB, HdfsSink.setInactiveBucketCheckInterval(10000L); HdfsSink.setInactiveBucketThreshold(10000L); Thanks a lot. Regards, Raja.