Hey- sorry I misunderstood your question. I don't think you're going to find an out-of-the-box solution for doing this with Sqoop. My suggestion is you'll need to schedule some ETL work in Map reduce, Pig, or Hive in order to separate out only the records your want to export, then call Sqoop to move the new file. I have a similar situation and Sqoop exports are really just a transport mechanism to move static data out of HDFS. Shaping that data is left to other tools.
From: Muthu Pandi <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, July 16, 2014 12:14 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Export only newly inserted values Yes i can use Oozie for the schedule but while export the whole data will be exported again and again. let say i had 10000 rows at 4:00 and i do export 10000 data will be exported after 15 Min data will be increased by 10500 so while exporting at 4:15 again 10500 data will be exported i need to export only the new data that is 500 data. Regards Muthupandi.K 9894594112 [Picture (Device Independent Bitmap)] On Tue, Jul 15, 2014 at 10:06 PM, Brenden Cobb <[email protected]<mailto:[email protected]>> wrote: You might want to look into creating an Oozie job to handle your periodic exports. From: Muthu Pandi <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, July 14, 2014 12:25 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Export only newly inserted values Yes but i have to export periodically say 15 mins for the same day Regards Muthupandi.K 9894594112 [Picture (Device Independent Bitmap)] On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <[email protected]<mailto:[email protected]>> wrote: If your hive table is partitioned by date (say year/month/day), then you can export data from a specific partition using hcatalog Thanks Venkat On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <[email protected]<mailto:[email protected]>> wrote: Dear All Am in an analytic project where all the data are fed into hive warehouse and am in need of exporting those values to mysql since the analytic on hive is too slow. Data will be coming to hive in timely fashion so i need to export only the new data not all the data using sqoop export kindly point me the direction. Regards Muthupandi.K [Picture (Device Independent Bitmap)] CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
