Hi Srinivas, I am sorry that you are running into these issues. I think you may be hitting [SQOOP-314]. We have recently fixed it on the trunk, so can you please try out a build from the trunk to see if it resolves your problem. You should run the export with the --batch option.
Also, to subscribe to sqoop-user@incubator.apache.org, you would first need to send a mail to incubator-sqoop-user-subscr...@apache.org and follow the instructions that come back. [SQOOP-314] https://issues.apache.org/jira/browse/SQOOP-314 Thanks, Arvind On Thu, Sep 8, 2011 at 8:30 PM, SRINIVAS SURASANI <vas...@gmail.com> wrote: > Arvind, > I have tried sending an reply to sqoop-user@incubator.apache.org but its > giving me failure notification. > > Sqoop export is hanging up after 100% of map is done. Even I have tried > with "sqoop.export.records.per.statement"=1 still after 100% of map, job > goes into idle condition.Importing into HDFS from Terradata is working > fine. > > Thanks, > Srini > On Wed, Aug 31, 2011 at 12:19 AM, SRINIVAS SURASANI <vas...@gmail.com> > wrote: >> >> Arvind, >> Here we have two scenarios. >> 1.The xml data as it is processed need to be loaded into Terradata ( this >> is for analysis of intra-day). So, we will be having around 3500 files/day >> to be loaded into Terrdata. This is like semi-real kind of loading. Duration >> between source landing(xml data into HDFS) and loading(csv files into >> Terradata) is couple of minutes. >> 2. As a batch. Meaning, proceesing entire day data together ( xml to csv >> using Pig code) and sending this single directory as a batch. >> >> The scenarios might change.. as the project is in early stages. >> Regards, >> Srini >> >> On Tue, Aug 30, 2011 at 11:48 PM, arv...@cloudera.com >> <arv...@cloudera.com> wrote: >>> >>> Thanks Srini. What kind of work load do you have? >>> >>> - Arvind >>> >>> On Tue, Aug 30, 2011 at 5:47 PM, SRINIVAS SURASANI <vas...@gmail.com> >>> wrote: >>> > Arvind, >>> > I'm working as contractor at MorganStanley. >>> > >>> > Srini >>> > On Tue, Aug 30, 2011 at 5:41 PM, arv...@cloudera.com >>> > <arv...@cloudera.com> >>> > wrote: >>> >> >>> >> Hi Srini, >>> >> >>> >> I am glad I was able to help. Do you mind sharing with me some more >>> >> details regarding how/where you are using Sqoop? I keep a track of the >>> >> various deployments in order to help prioritize the feature >>> >> development so that it is aligned with what the users need rather than >>> >> what we want to do first. >>> >> >>> >> It will be great if you can give me an idea of which company this >>> >> project was for and what was the high-level workload like, as long as >>> >> it is public information. >>> >> >>> >> Thanks, >>> >> Arvind >>> >> >>> >> >>> >> ---------- Forwarded message ---------- >>> >> From: SRINIVAS SURASANI <vas...@gmail.com> >>> >> Date: Mon, Aug 29, 2011 at 3:06 PM >>> >> Subject: Re: [sqoop-user] Sqoop-with Terradata >>> >> To: sqoop-u...@cloudera.org >>> >> >>> >> >>> >> Arvind, >>> >> >>> >> Thank-you very much. I'm able to connect to Terradata. >>> >> >>> >> >>> >> Srini >>> >> >>> >> On Mon, Aug 29, 2011 at 2:58 PM, arv...@cloudera.com >>> >> <arv...@cloudera.com> wrote: >>> >> > >>> >> > This may be related to not having the GSS configuration classes in >>> >> > your classpath. I would suggest you add tdgssconfig.jar to the >>> >> > classpath and try again. >>> >> > >>> >> > Thanks, >>> >> > Arvind >>> >> > >>> >> > On Mon, Aug 29, 2011 at 11:19 AM, SRINIVAS SURASANI >>> >> > <vas...@gmail.com> >>> >> > wrote: >>> >> > > Arvind, >>> >> > > >>> >> > > I have subscribed at sqoop-user@incubator.apache.org and posted >>> >> > > Question . >>> >> > > Sorry for the inconvinence from my end, since I'm close to >>> >> > > deadline >>> >> > > I'm >>> >> > > taking your valuable time. >>> >> > > >>> >> > > sqoop list-tables --driver com.teradata.jdbc.TeraDriver --connect >>> >> > > jdbc:teradata://PKTD/E1_CMS_WORK --username srini -P >>> >> > > Iam getting the following error: >>> >> > > 11/08/29 13:08:03 INFO manager.SqlManager: Using default fetchSize >>> >> > > of >>> >> > > 1000 >>> >> > > GSSException: Failure unspecified at GSS-API level (Mechanism >>> >> > > level: >>> >> > > UserFile parameter null) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssParseXml.<init>(DashoA1*..) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssConfigApi.<init>(DashoA1*..) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.<init>(DashoA1*..) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.getInstance(DashoA1*..) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:612) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getConfig(GenericTeraEncrypt.java:630) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getUserNameForOid(GenericTeraEncrypt.java:723) >>> >> > > at >>> >> > > com.teradata.jdbc.AuthMechanism.<init>(AuthMechanism.java:50) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericInitDBConfigState.action(GenericInitDBConfigState.java:105) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:49) >>> >> > > at >>> >> > > com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:199) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc_3.ifjdbc_4.TeraLocalConnection.<init>(TeraLocalConnection.java:95) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:54) >>> >> > > at >>> >> > > com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:217) >>> >> > > at >>> >> > > com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:150) >>> >> > > at >>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:582) >>> >> > > at >>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:185) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.cloudera.sqoop.manager.SqlManager.makeConnection(SqlManager.java:643) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.cloudera.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:51) >>> >> > > at >>> >> > > >>> >> > > com.cloudera.sqoop.manager.SqlManager.listTables(SqlManager.java:270) >>> >> > > at >>> >> > > com.cloudera.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49) >>> >> > > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144) >>> >> > > at >>> >> > > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> >> > > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180) >>> >> > > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219) >>> >> > > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228) >>> >> > > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237) >>> >> > > 11/08/29 13:08:04 ERROR sqoop.Sqoop: Got exception running Sqoop: >>> >> > > java.lang.NullPointerException >>> >> > > java.lang.NullPointerException >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssConfigApi.GetMechanisms(DashoA1*..) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.<init>(DashoA1*..) >>> >> > > at >>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.getInstance(DashoA1*..) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:612) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getConfig(GenericTeraEncrypt.java:630) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getUserNameForOid(GenericTeraEncrypt.java:723) >>> >> > > at >>> >> > > com.teradata.jdbc.AuthMechanism.<init>(AuthMechanism.java:50) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericInitDBConfigState.action(GenericInitDBConfigState.java:105) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:49) >>> >> > > at >>> >> > > com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:199) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc_3.ifjdbc_4.TeraLocalConnection.<init>(TeraLocalConnection.java:95) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:54) >>> >> > > at >>> >> > > com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:217) >>> >> > > at >>> >> > > com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:150) >>> >> > > at >>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:582) >>> >> > > at >>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:185) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.cloudera.sqoop.manager.SqlManager.makeConnection(SqlManager.java:643) >>> >> > > at >>> >> > > >>> >> > > >>> >> > > com.cloudera.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:51) >>> >> > > at >>> >> > > >>> >> > > com.cloudera.sqoop.manager.SqlManager.listTables(SqlManager.java:270) >>> >> > > at >>> >> > > com.cloudera.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49) >>> >> > > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144) >>> >> > > at >>> >> > > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> >> > > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180) >>> >> > > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219) >>> >> > > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228) >>> >> > > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237) >>> >> > > Thanks, >>> >> > > Srini >>> >> > > >>> >> > > >>> >> > > >>> >> > > On Mon, Aug 29, 2011 at 12:58 PM, arv...@cloudera.com >>> >> > > <arv...@cloudera.com> >>> >> > > wrote: >>> >> > >> >>> >> > >> [Please subscribe and respond to sqoop-user@incubator.apache.org] >>> >> > >> >>> >> > >> Please use HADOOP_CLASSPATH instead of CLASSPATH. Also, in order >>> >> > >> to >>> >> > >> use the generic JDBC connector, you will have to specify the >>> >> > >> driver >>> >> > >> class explicitly via the command line option --driver >>> >> > >> com.teradata.jdbc.TeraDriver. >>> >> > >> >>> >> > >> Thanks, >>> >> > >> Arvind >>> >> > >> >>> >> > >> On Mon, Aug 29, 2011 at 9:53 AM, SRINIVAS SURASANI >>> >> > >> <vas...@gmail.com> >>> >> > >> wrote: >>> >> > >> > Arvind, >>> >> > >> > I have set the classpath to teradata4.jar [ not placed the >>> >> > >> > teradata4.jar >>> >> > >> > in >>> >> > >> > sqoop lib, as I dont have permissions]. >>> >> > >> > I'm getting the following error >>> >> > >> > >>> >> > >> > sqoop list-tables --connect jdbc:teradata://PKTD/E1_CMS_WORK >>> >> > >> > --username >>> >> > >> > srini -P >>> >> > >> > ERROR: tool.BaseSqoopTool: Got error creating database manager: >>> >> > >> > java.io.IOexception: No manager for connect string: >>> >> > >> > jdbc:teradata:///PKTD/E1_CMS_WORK >>> >> > >> > at >>> >> > >> > com.cloudera.sqoop.ConnFactory.getManager(ConnFactory.java:119) >>> >> > >> > at >>> >> > >> > >>> >> > >> > com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:186) >>> >> > >> > ... >>> >> > >> > ... >>> >> > >> > Thanks, >>> >> > >> > Srini >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > On Mon, Aug 29, 2011 at 8:52 AM, SRINIVAS SURASANI >>> >> > >> > <vas...@gmail.com> >>> >> > >> > wrote: >>> >> > >> >> >>> >> > >> >> Thanks-a-lot Arvind. >>> >> > >> >> >>> >> > >> >> On Mon, Aug 29, 2011 at 8:45 AM, arv...@cloudera.com >>> >> > >> >> <arv...@cloudera.com> >>> >> > >> >> wrote: >>> >> > >> >>> >>> >> > >> >>> [Moving the thread to sqoop-user@incubator.apache.org] >>> >> > >> >>> >>> >> > >> >>> Hi Srini, >>> >> > >> >>> >>> >> > >> >>> You should be able to use the generic JDBC connector to >>> >> > >> >>> import/export >>> >> > >> >>> from Teradata. There is also a specialized connector that is >>> >> > >> >>> available >>> >> > >> >>> for use with Teradata if you are interested. This connector >>> >> > >> >>> is >>> >> > >> >>> not a >>> >> > >> >>> part of Sqoop and can be obtained from Cloudera by going to: >>> >> > >> >>> >>> >> > >> >>> http://www.cloudera.com/partners/connectors/ >>> >> > >> >>> >>> >> > >> >>> Thanks, >>> >> > >> >>> Arvind >>> >> > >> >>> >>> >> > >> >>> On Mon, Aug 29, 2011 at 8:17 AM, SRINIVAS SURASANI >>> >> > >> >>> <vas...@gmail.com> >>> >> > >> >>> wrote: >>> >> > >> >>> > I have csv file in hadoop and looking to load into >>> >> > >> >>> > Teradata. I >>> >> > >> >>> > was >>> >> > >> >>> > wondering does the sqoop works with Terradata.(with JDBC >>> >> > >> >>> > jar >>> >> > >> >>> > placing >>> >> > >> >>> > in sqoop lib dir). >>> >> > >> >>> > >>> >> > >> >>> > Regards >>> >> > >> >>> > Srini >>> >> > >> >>> > >>> >> > >> >>> > -- >>> >> > >> >>> > NOTE: The mailing list sqoop-u...@cloudera.org is >>> >> > >> >>> > deprecated in >>> >> > >> >>> > favor >>> >> > >> >>> > of Apache Sqoop mailing list >>> >> > >> >>> > sqoop-user@incubator.apache.org. >>> >> > >> >>> > Please >>> >> > >> >>> > subscribe to it by sending an email to >>> >> > >> >>> > incubator-sqoop-user-subscr...@apache.org. >>> >> > >> >>> > >>> >> > >> >>> >>> >> > >> >>> -- >>> >> > >> >>> NOTE: The mailing list sqoop-u...@cloudera.org is deprecated >>> >> > >> >>> in >>> >> > >> >>> favor >>> >> > >> >>> of >>> >> > >> >>> Apache Sqoop mailing list sqoop-user@incubator.apache.org. >>> >> > >> >>> Please >>> >> > >> >>> subscribe >>> >> > >> >>> to it by sending an email to >>> >> > >> >>> incubator-sqoop-user-subscr...@apache.org. >>> >> > >> >> >>> >> > >> > >>> >> > >> > -- >>> >> > >> > NOTE: The mailing list sqoop-u...@cloudera.org is deprecated in >>> >> > >> > favor of >>> >> > >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. >>> >> > >> > Please >>> >> > >> > subscribe >>> >> > >> > to it by sending an email to >>> >> > >> > incubator-sqoop-user-subscr...@apache.org. >>> >> > >> > >>> >> > >> >>> >> > >> -- >>> >> > >> NOTE: The mailing list sqoop-u...@cloudera.org is deprecated in >>> >> > >> favor >>> >> > >> of >>> >> > >> Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please >>> >> > >> subscribe >>> >> > >> to it by sending an email to >>> >> > >> incubator-sqoop-user-subscr...@apache.org. >>> >> > > >>> >> > > -- >>> >> > > NOTE: The mailing list sqoop-u...@cloudera.org is deprecated in >>> >> > > favor >>> >> > > of >>> >> > > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please >>> >> > > subscribe >>> >> > > to it by sending an email to >>> >> > > incubator-sqoop-user-subscr...@apache.org. >>> >> > > >>> >> > >>> >> > -- >>> >> > NOTE: The mailing list sqoop-u...@cloudera.org is deprecated in >>> >> > favor of >>> >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please >>> >> > subscribe >>> >> > to it by sending an email to >>> >> > incubator-sqoop-user-subscr...@apache.org. >>> >> >>> >> -- >>> >> NOTE: The mailing list sqoop-u...@cloudera.org is deprecated in favor >>> >> of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please >>> >> subscribe to it by sending an email to >>> >> incubator-sqoop-user-subscr...@apache.org. >>> > >>> > >> > >
