Thanks Matt On Sat, Apr 21, 2018 at 12:43 AM, Matt Sicker <[email protected]> wrote:
> It's not a Flume native solution, but an alternative I used in the past > was Kafka Connect using the HDFS connector plugin. That plugin provides > configuration regarding how often to roll over Avro files. > > On 20 April 2018 at 13:49, Nitin Kumar <[email protected]> wrote: > >> Hi All, >> >> I am using Flume v1.8 in which Flume agent comprises of Kafka Channel & >> HDFS Sink. >> I am able to write data in Avro file on HDFS into a external HIVE table, >> but the problem is whenever Flume gets restarted it closes that file and >> open a new file because of which I can see many small files. (Data is >> partition on the basis of date) >> >> Can't Flume append to existing file to avoid creation of new file? >> Also, how can I solve this problem which leads to creation of too many >> small files? >> >> Any help would be appreciated. >> >> -- >> >> *Regards,Nitin Kumar* >> > > > > -- > Matt Sicker <[email protected]> > -- *Regards,Nitin Kumar Choudhary*
