Hi, Assuming you are using Avro then you might want to take a look at: https://github.com/confluentinc/camus
If not you can still adopt the pattern used to get data into HDFS. We use something similar at my place of employment. Thanks, Damian On 6 November 2015 at 08:46, Woodbine, Kevin <[email protected]> wrote: > Dear Sirs, > > I'm working on a development project for my company that is looking at the > possibility of live streaming transaction data from our Oracle databases > into Hadoop HDFS. Doing some research I've been looking into leveraging > the power of Kafka to form part of the solution and need some a pointer > which I hope you can help me with. > > We use Goldengate to stream data into a Kafka topic and can see this data > when running to command: > > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test > --from-beginning > > My question is how would I get Kafka to write this data from the topic > into a HDFS directory? > > Any help you can provide will be gratefully received. > > Many thanks > > Regards > > Kevin Woodbine > * (CUCI) 01784 736039 > * 07990 563 530 > [email protected]<mailto:[email protected]> > > > _____________________________________________________________________ > The information contained in or attached to this email is intended only > for the use of the individual or entity to which it is addressed. If you > are not the intended recipient, or a person responsible for delivering it > to the intended recipient, you are not authorised to and must not disclose, > copy, distribute, or retain this message or any part of it. It may contain > information which is confidential and/or covered by legal professional or > other privilege (or other rules or laws with similar effect in > jurisdictions outside England and Wales). > > The views expressed in this email are not necessarily the views of > Centrica plc, and the company, its directors, officers or employees make no > representation or accept any liability for its accuracy or completeness > unless expressly stated to the contrary. > > PH Jones is a trading name of British Gas Social Housing Limited. British > Gas Social Housing Limited (company no: 01026007), British Gas Trading > Limited (company no: 03078711), British Gas Services Limited (company no: > 3141243), British Gas Insurance Limited (company no: 06608316), British Gas > New Heating Limited (company no: 06723244), British Gas Services > (Commercial) Limited (company no: 07385984) and Centrica Energy (Trading) > Limited (company no: 02877397) are all wholly owned subsidiaries of > Centrica plc (company no: 3033654). Each company is registered in England > and Wales with a registered office at Millstream, Maidenhead Road, Windsor, > Berkshire SL4 5GD. > > British Gas Insurance Limited is authorised by the Prudential Regulation > Authority and regulated by the Financial Conduct Authority and the > Prudential Regulation Authority. British Gas Services Limited and Centrica > Energy (Trading) Limited are authorised and regulated by the Financial > Conduct Authority. British Gas Trading Limited is an appointed > representative of British Gas Services Limited which is authorised and > regulated by the Financial Conduct Authority.
