Re: GG kafka topic in avro format to NiFi

Colin, You would have not seen this error before from the Kafka processor you were using as it would not be doing any deserialization. You would have seen that error downstream in the LookupRecord processor you had. Just wanted to clarify that point. That you're seeing the error now with

Re: GG kafka topic in avro format to NiFi

Hi, I configured ConsumeKafkaRecord and set schema.name to the value. I think something had preventing me from editing it's value previously. Anyhow I have what looks like a similar error message: 2018-03-22 09:24:24,990 ERROR [Timer-Driven Process Thread-6]

Re: GG kafka topic in avro format to NiFi

Hello, You don’t need to use a properties file. In your AvroRecordReader, just change the Schema Name property from ${schema.name} to the actual name of the schema you want to use from the schema registry. It just means that the record reader can only be used with one schema now, rather than

Re: GG kafka topic in avro format to NiFi

Hi Joe, Thanks for the suggestion. I started by using the ConsumeKafkaRecord0_10. But I had read the only way to configure the schema.name was via a properties file, which I read also required a restart of NiFi.

Re: GG kafka topic in avro format to NiFi

Colin, You're using the ConsumeKafka processors. Given that this is avro data for which you have a schema/etc.. I strongly recommend you use ConsumeKafkaRecord0_10... In that you get to specify the record reader/writer you'll need. You will also see dramatically higher performance. Lets get

Re: GG kafka topic in avro format to NiFi

Hi Joe, I don't believe the Avro schema included, and expect they are the data portion... I think that's why I need to use the avsc file mentioned above... On Wed, Mar 21, 2018 at 6:19 PM, Joe Witt wrote: > Can you share a template of your process group? > > Do the messages

Re: GG kafka topic in avro format to NiFi

Can you share a template of your process group? Do the messages in Kafka have the Avro schema included in them or are they just the data portion of the record? On Wed, Mar 21, 2018 at 9:16 PM, Colin Williams wrote: > I have an avro avsc file for a table with a

GG kafka topic in avro format to NiFi

I have an avro avsc file for a table with a definition like: