RE: KafkaSimpleJsonExtractor

2018-04-19 Thread Kidwell, Jack
nd why this was not done earlier. I require some time to look deeper into it, hope will take some time during the weekend. On Sat, Apr 14, 2018 at 2:41 AM, Kidwell, Jack <jack_kidwel...@comcast.com<mailto:jack_kidwel...@comcast.com>> wrote: Our correspondence helps with und

RE: KafkaSimpleJsonExtractor

2018-04-13 Thread Kidwell, Jack
can use KafkaSimpleJsonExtractor which was asked at the beginning. So to conclude what I understand is that you don't want the data to be stored in the byte array, you require json (or other format). On Thu, Apr 12, 2018 at 8:09 PM, Kidwell, Jack <jack_kidwel...@comcast.

RE: KafkaSimpleJsonExtractor

2018-04-12 Thread Kidwell, Jack
er. On Thu, Apr 12, 2018 at 6:47 PM, Kidwell, Jack <jack_kidwel...@comcast.com<mailto:jack_kidwel...@comcast.com>> wrote: We want to extract both key and value from a Kafka message and publish the combined information to HDFS. Keys contain numeric ids for the strings contained i

RE: KafkaSimpleJsonExtractor

2018-04-12 Thread Kidwell, Jack
to see the keys/values rendered in the logs while the extraction is being done? Why can't you have the other subscriber to the kafka which will render the values rather than KafkaExtractor implementation rendering the same? On Wed, Apr 11, 2018 at 11:17 PM, Kidwell, Jack <jack_kid

Kafka09ConsumerClient.java

2018-04-11 Thread Kidwell, Jack
Hi, In gobblin-distribution-0.10.0.tar.gz no jar contains the classes defined in Kafka09ConsumerClient.java. Is this expected? Or must I build the jar from source? Thanks.

KafkaSimpleJsonExtractor

2018-04-11 Thread Kidwell, Jack
Hi, Using gobblin_0.10.0, we want to use module, KafkaSimpleJsonExtractor.java in order to see both kafka record keys and values. How does one configure a job to achieve it?