Ali - I can confirm that all formats are supported for Put/Get Kafka processors. The interaction with Kafka as far as the messages are concerned is just 'bytes'. The reason it isn't working well is due to the bug I'm mentioning and that is a NiFi bug (not Kafka). It is fixed and we'll have a release for you asap but if you want to build that branch then you're in great shape right away.
Thanks On Wed, Apr 6, 2016 at 10:24 AM, Ali Karray <[email protected]> wrote: > I see the patch in Jira. > You confirm that avro format is supported for putKafka and getKafka processor > ? > > Thanks > Ali > > On Wed, Apr 6, 2016 at 4:14 PM, Joe Witt <[email protected]> wrote: >> Ali >> >> It is very possible you're being bitten by a bug we have with >> PutKafka. It is fixed in this branch which we're in the release >> motions for right now. >> >> https://github.com/apache/nifi/tree/support/nifi-0.6.x >> >> Thanks >> Joe >> >> On Wed, Apr 6, 2016 at 10:07 AM, Ali Karray <[email protected]> wrote: >>> I'm using the last version 0.6 of Nifi and the 0.9 kafka version >>> When I use plain csv files, all is working. >>> >>> Thanks, >>> Ali >>> >>> >>> On Wed, Apr 6, 2016 at 3:23 PM, Bryan Bende <[email protected]> wrote: >>>> >>>> Hello, >>>> >>>> What version of Apache NiFi are you using? >>>> >>>> Also, lets take Avro and kafka-avro-console-consumer out of the mix for a >>>> minute, what happens if you used PutKafka to send some plain text to >>>> Kafka, and then tried to consumer from that topic with GetKafka? >>>> You mentioned this didn't work, but what happened? was there an error >>>> message? did it just receive 0 messages from Kafka? >>>> >>>> Thanks, >>>> >>>> Bryan >>>> >>>> On Wed, Apr 6, 2016 at 6:10 AM, Ali Karray <[email protected]> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm exploring integration possibility of nifi processors with kafka. >>>>> I use PutKafka to puch avro generated flow from ExecuteSQL. >>>>> I verify that the topic created in Kafka is alimented with data using >>>>> kafka-console-consumer bash command. >>>>> Then I try to use getKafka processor to pull all the Data stored in >>>>> the kafka topic and store all that data in a local File using the >>>>> PutFile processor. I realize that the getKafka processor don't pull >>>>> the topic content. >>>>> >>>>> I verify if all is ok with kafka-avro-console-consumer (confluent >>>>> command), I get this exception : >>>>> org.apache.kafka.common.errors.SerializationException: Error >>>>> deserializing Avro message for id -1 >>>>> Caused by: org.apache.kafka.common.errors.SerializationException: >>>>> Unknown magic byte! >>>>> >>>>> I tested also getKafka from a basic topic format and it doesn't work for >>>>> me. >>>>> >>>>> So I have some questions : >>>>> How can I deal with avro format using getKafka anf putKafka processor ? >>>>> Any idea to resolve getKafka processor problem to pull content from a >>>>> kafka Topic ? >>>>> >>>>> >>>>> Thank you in advance >>>>> >>>>> A. Karray >>>> >>>>
