For writing the data to the HDFS you need to take a look at the Writer
Implementation, please look for it. The Extractor is used in pulling the
data which is thereafter passed through various stages before it is stored
via the Writer.
What I mean by the other subscriber was to add one more consumer which
will  read the key/value e.g kafka-console-consumer.sh which comes with the
kafka.
I was not earlier aware that you wanted to store the data in HFDS as you
had given the reference of the Extractor so it was not clear to me.
Please take a look at the Kafka to HDFS writer and see if that helps, in
case it is not doing what is exactly required by you then you may have to
plugin the customized writer.


On Thu, Apr 12, 2018 at 6:47 PM, Kidwell, Jack <jack_kidwel...@comcast.com>
wrote:

> We want to extract both key and value from a Kafka message and publish the
> combined information to HDFS. Keys contain numeric ids for the strings
> contained in values.
>
>
>
> Please explain “other subscriber”. Are you proposing a different Kafka
> message structure?
>
>
>
> KafkaSimpleJsonExtractor.java caught my attention because it extracts both
> keys and values and puts them in decodedRecord.
>
>
>
> *From:* Vicky Kak [mailto:vicky....@gmail.com]
> *Sent:* Thursday, April 12, 2018 1:17 AM
> *To:* user@gobblin.incubator.apache.org
> *Subject:* Re: KafkaSimpleJsonExtractor
>
>
>
> I am not sure what you are asking for. Do you want to see the keys/values
> rendered in the logs while the extraction is being done?
>
> Why can't you have the other subscriber to the kafka which will render the
> values rather than KafkaExtractor implementation rendering the same?
>
>
>
>
>
> On Wed, Apr 11, 2018 at 11:17 PM, Kidwell, Jack <
> jack_kidwel...@comcast.com> wrote:
>
> Hi,
>
>
>
> Using gobblin_0.10.0, we want to use module, KafkaSimpleJsonExtractor.java
> in order to see both kafka record keys and values.
>
>
>
> How does one configure a job to achieve it?
>
>
>
>
>

Reply via email to