Re: sqoop import big table with no integer key

2018-08-14 Thread Nicolas Paris
hi Raymond Postgresql also have a direct mode that don't need any split by. Direct mode is usually a good idea since it does not load the source database with multiple threads and as better performance 2018-08-14 10:42 GMT+02:00 Szabolcs Vasas : > Hi Raymond, > > Yes, I can confirm that

Re: sqoop import big table with no integer key

2018-08-14 Thread Szabolcs Vasas
Hi Raymond, Yes, I can confirm that splitting by a string field can cause issues in Sqoop that is why the org.apache.sqoop.splitter.allow_text_splitter property was introduced (see: SQOOP-2910). Which RDBMS do you use? If you use Oracle then you are lucky because the Oracle direct connector does

sqoop import big table with no integer key

2018-07-11 Thread Raymond Xie
Good day, In my situation my table has billion rows, it doesn't come with an integer column as its key, that means if I use sqoop to do the import (into hive), I would not be able to use multiple mapper. As table's size is big, it is not realistic to add an extra new integer field to it. I do