Hi Abhijeet --, Thanks for the information. I have one more question. Is the exports is done always with one mapper? ( entering into table deadlocks if number of mappers set to more than one ). Also, FYI: I have observed the default number of rows inserted is 1000.
Thanks, -- Srinvas On Sat, Jan 28, 2012 at 10:12 AM, abhijeet gaikwad <abygaikwa...@gmail.com>wrote: > Hi Srinivas, > Haven't played with Teradata Connector, but in general there are two > properties that drive insertions (SQOOP Export) in a table - namely: > > 1. "sqoop.export.records.per.statement" : This property is used to > specify the number of records/rows to be inserted using a single > INSERT statement. Default value is 100. > 2. "sqoop.export.statements.per.transaction" : This property is used > to specify the number the insert statements before a commit is fired - > which you can call batch size. Default value is 100. > > You can use -D hadoop argument to specify these properties at command > line. E.g. -Dsqoop.export.statements.per.transaction=50 > > NOTE: Make sure you use this argument(-D) before using any of the > SQOOP tool specific arguments. See SQOOP User Guide for more details. > > Thanks, > Abhijeet Gaikwad > > On 1/26/12, Srinivas Surasani <vas...@gmail.com> wrote: > > Kathleen, > > > > Any information on below request. > > > > Hi All, > > > > I'm working on Hadoop CDH3 U0 and Sqoop CDH3 U2. > > > > I'm trying to export csv files from HDFS to Teradata, it works well with > > setting mapper to "1" ( with batch loading of 1000 records at a time ). > > when I tried increasing the number of mappers to more than one I'm > getting > > the following error. Also, is it possible to configure batch size at the > > time of export ( from the command line)?? > > > > > > sqoop export --verbose --driver com.teradata.jdbc.TeraDriver > > --connect jdbc:teradata://xxxx/database=xxxx --username xxxxx --password > > xxxxx --table xxxx --export-dir /user/surasani/10minutes.txt > > --fields-terminated-by '|' -m 4 --batch > > > > 12/01/24 16:17:21 INFO mapred.JobClient: map 3% reduce 0% > > 12/01/24 16:17:48 INFO mapred.JobClient: Task Id : > > attempt_201112211106_68553_m_000001_2, Status : FAILED > > *java.io.IOException: java.sql.BatchUpdateException: [Teradata Database] > > [TeraJDBC 13.00.00.07] [Error 2631] [SQLState 40001] Transaction ABORTed > > due to DeadLock*. > > > > Srinivas -- > > > > On Wed, Jan 25, 2012 at 8:01 PM, Kathleen Ting <kathl...@cloudera.com > >wrote: > > > >> Srinivas, as it happens, the Cloudera Connector for Teradata supports > >> staging tables. It is freely available here: > >> > https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Teradata+Download > >> . > >> > >> Regards, Kathleen > >> > >> On Wed, Jan 25, 2012 at 3:36 PM, Srinivas Surasani > >> <vas...@gmail.com>wrote: > >> > >>> Hi Kathleen, > >>> > >>> Same issue with Teradata. > >>> > >>> > >>> Srinivas -- > >>> > >>> > >>> On Mon, Jan 23, 2012 at 8:26 PM, Kathleen Ting > >>> <kathl...@cloudera.com>wrote: > >>> > >>>> Hi Weihua - > >>>> > >>>> Unfortunately, the generic jdbc manager does not support staging. > >>>> As a result, I've filed > >>>> https://issues.apache.org/jira/browse/SQOOP-431on your behalf. > >>>> > >>>> Regards, Kathleen > >>>> > >>>> > >>>> On Mon, Jan 23, 2012 at 3:10 PM, Weihua Zhu <w...@adconion.com> > wrote: > >>>> > >>>>> Hi Guys, > >>>>> > >>>>> Good afternoon! > >>>>> I have a question. I was trying to sqoop exporting from hdfs to > >>>>> postgresql, using --staging-table options due to transactions > >>>>> consideration. But it gives me error below. > >>>>> I am wondering if the staging_able is supported for > >>>>> GenericJdbcManager? if not, what kind of manager should I use? > >>>>> Thanks very much! > >>>>> > >>>>> -Weihua > >>>>> > >>>>> error message: > >>>>> > >>>>> 12/01/23 15:00:39 ERROR tool.ExportTool: Error during export: The > >>>>> active connection manager > (org.apache.sqoop.manager.GenericJdbcManager) > >>>>> does not support staging of data for export. Please retry without > >>>>> specifying the --staging-table option. > >>>>> > >>>>> > >>>>> > >>>> > >>> > >> > > >