Nifi 1.9 FlowFile Repository failed to update , java.nio.file.FileAlreadyExistsException

2019-03-08 Thread Viking K
Hi, I recently upgraded our cluster to 1.9 from 1.7.1 and I have seen some weird behaviors relating to the flowfile repo. Since upgrading i have had at-least 3 occurrences of the following error . nifi-app_2019-03-07_15.0.log:Caused by: java.nio.file.FileAlreadyExistsException:

Re: How to consume avro messages with schema reference from Kafka, into large flowfiles

2018-12-12 Thread Viking K
As Bryan said knowing some more about your setup and expectations would help. But I can give some information from my own evaluation of a similar setup. I needed a new data pipeline into a data lake, the setup of the downstream systems were like this. Downstream components: * Multiple

Re: ConsumeKafkaRecord won't pull new events from Kafka

2018-11-13 Thread Viking K
On Tue, Nov 13, 2018 at 12:24 PM Viking K wrote: > > Mike, are we talking about complete avro messages or bare records (schema is > contained inside the avro file or do they use a schema registry) > > From my own testing the ConsumeKafkaRecord tries to bundle in the incoming >

Re: ConsumeKafkaRecord won't pull new events from Kafka

2018-11-13 Thread Viking K
Mike, are we talking about complete avro messages or bare records (schema is contained inside the avro file or do they use a schema registry) >From my own testing the ConsumeKafkaRecord tries to bundle in the incoming >messages to create larger flow files. Do you use any Kafka headers in the