Hi Szymon, Unfortunately Sqoop can't yet read partition column information out of the Hive table. So you'll need to export each partition individually. You can probably get your system to work right by doing these two commands:
sqoop --connect jdbc:mysql://test-db.gadu/crunchers --username crunchers --table tb3 --export-dir /user/hive/warehouse/tb1/day=2010-04-01 mysql -e "update tb3 set day='2010-04-01' where day is null" crunchers and then repeat this for each partition subdir. If you're clever with bash and awk, you can probably get this running in a loop in a script. If that doesn't work, then you might need to first import into a mysql tmp table that only contains the first two columns, and then do an INSERT INTO... SELECT into the true (3 column) table followed by the UPDATE WHERE day IS NULL statement. Sorry this seems like more of a hassle than it should be. This sort of feature is really important. Can I encourage you to try at writing a patch, if you're handy? :) Send me an email off-list if you want to help. - Aaron 2010/6/18 Szymon Gwóźdź <[email protected]> > Hi! > > I have table tb1 defined by: > CREATE TABLE tb1(user int, counter int) PARTITIONED BY (day string) STORED > AS TEXTFILE > > I want to export data from this table into mysql table defined by: > CREATE TABLE tb2(user int, counter int, day string) > > I've tried to use Sqoop in order to do this but Sqoop doesn't allow to > export directory with partitions - while trying: > sqoop --connect jdbc:mysql://test-db.gadu/crunchers --username crunchers > --password RydBert3 --table tb2 --export-dir /user/hive/warehouse/tb1 > I get: > 10/06/18 13:55:03 WARN mapreduce.ExportJob: IOException checking > SequenceFile header: java.io.IOException: Cannot open filename > /user/hive/warehouse/tb1/day=2010-04-01 > It is possible to do something like this: > sqoop --connect jdbc:mysql://test-db.gadu/crunchers --username crunchers > --password RydBert3 --table tb3 --export-dir > /user/hive/warehouse/tb1/day=2010-04-01 > using mysql table defined by: > CREATE TABLE tb3(user int, counter int), > but it is not the thing I want to do, because I want to have "day" column > in mysql table. > > Does someone know what to do in order to load this data into mysql? > > Cheers, > Szymon Gwóźdź >
