[jira] [Commented] (SQOOP-3022) sqoop export for Oracle generates tremendous amounts of redo logs
[ https://issues.apache.org/jira/browse/SQOOP-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570633#comment-15570633 ] David Robson commented on SQOOP-3022: - I haven't tested all the various scenarios in a while but basically if you use partitioning by setting "oraoop.partitioned" to "TRUE" it should create a table per mapper and do direct path inserts. http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_partitioning If you don't want to use partitioning but want to disable redo logs then you can set "oraoop.oracle.append.values.hint.usage" to "ON" which will use the APPEND_VALUES hint for the insert statements into the target table. http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_oraoop_oracle_append_values_hint_usage So I think it already does what you require so we should be able to close this issue? > sqoop export for Oracle generates tremendous amounts of redo logs > - > > Key: SQOOP-3022 > URL: https://issues.apache.org/jira/browse/SQOOP-3022 > Project: Sqoop > Issue Type: Bug > Components: codegen, connectors, connectors/oracle >Affects Versions: 1.4.3, 1.4.4, 1.4.5, 1.4.6 >Reporter: Ruslan Dautkhanov > Labels: export, oracle > > Sqoop export for Oracle generates tremendous amounts of redo logs (comparable > to export size or more). > We have put target tables in nologgin mode, but Oracle will still generate > redo logs unless +APPEND Oracle insert hint is used. > See https://oracle-base.com/articles/misc/append-hint for examples. > Please add an option for sqoop to generate insert statements in Oracle with > APPEND statement. Our databases are swamped with redo/archived logs whenever > we sqoop data to them. This is easily avoidable. And from business > prospective sqooping to staging tables in nologgin mode is totally fine. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-2737) Cannot import table from Oracle with column with spaces in name
[
https://issues.apache.org/jira/browse/SQOOP-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051886#comment-15051886
]
David Robson commented on SQOOP-2737:
-
Can you please try the patch for
https://issues.apache.org/jira/browse/SQOOP-2723
I have fixed the code to quote the Oracle columns so it should also resolve the
issue with spaces. This is for the direct connector (using --direct option).
You'll need to build the current branch. Or if you need help I can do this for
you if you let me know what version of Hadoop you are running?
> Cannot import table from Oracle with column with spaces in name
> ---
>
> Key: SQOOP-2737
> URL: https://issues.apache.org/jira/browse/SQOOP-2737
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/oracle
>Affects Versions: 1.4.5
>Reporter: Jakub Havlík
>Priority: Critical
>
> When importing database containing table(s) with spaces in column name(s)
> following error is displayed:
> {code}
> Error: java.io.IOException: SQLException in nextKeyValue
> at
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLSyntaxErrorException: ORA-00923: FROM keyword not
> found where expected
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
> at
> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
> at
> oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:886)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
> at
> oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
> at
> oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3657)
> at
> oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1495)
> at
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
> at
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
> ... 11 more
> {code}
> As a enterprise user I would expect that I can import tables with spaces in a
> column name with no issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SQOOP-2729) Sqoop2: Integration Tests: In ShortStories.java we have a varchar(10000) column, which is too long for oracle
[ https://issues.apache.org/jira/browse/SQOOP-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047703#comment-15047703 ] David Robson commented on SQOOP-2729: - FYI you can work around this - but only if running Oracle 12c. So I think not using more than 4000 chars is the easier solution. https://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623 > Sqoop2: Integration Tests: In ShortStories.java we have a varchar(1) > column, which is too long for oracle > - > > Key: SQOOP-2729 > URL: https://issues.apache.org/jira/browse/SQOOP-2729 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.99.6 >Reporter: Abraham Fine >Assignee: Abraham Fine > Attachments: SQOOP-2729.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-2723) Oracle connector not working with lowercase columns
[ https://issues.apache.org/jira/browse/SQOOP-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047628#comment-15047628 ] David Robson commented on SQOOP-2723: - Thanks Jarcec. I had put it on review board too but forgot to link it! I think those failures are unrelated to anything I have changed? > Oracle connector not working with lowercase columns > --- > > Key: SQOOP-2723 > URL: https://issues.apache.org/jira/browse/SQOOP-2723 > Project: Sqoop > Issue Type: Bug > Components: connectors/oracle >Affects Versions: 1.4.6 >Reporter: David Robson >Assignee: David Robson > Fix For: 1.4.7 > > Attachments: SQOOP-2723.patch > > > For example the following table does not work for imports or exports: > CREATE TABLE "TestV500"."Xy" >( "a" NVARCHAR2(255) DEFAULT NULL, > "x" NUMBER(10,0) DEFAULT NULL, > "y" NUMBER(10,0) DEFAULT NULL >); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-2723) Oracle connector not working with lowercase columns
[ https://issues.apache.org/jira/browse/SQOOP-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-2723: Attachment: SQOOP-2723.patch > Oracle connector not working with lowercase columns > --- > > Key: SQOOP-2723 > URL: https://issues.apache.org/jira/browse/SQOOP-2723 > Project: Sqoop > Issue Type: Bug > Components: connectors/oracle >Affects Versions: 1.4.6 >Reporter: David Robson >Assignee: David Robson > Fix For: 1.4.7 > > Attachments: SQOOP-2723.patch > > > For example the following table does not work for imports or exports: > CREATE TABLE "TestV500"."Xy" >( "a" NVARCHAR2(255) DEFAULT NULL, > "x" NUMBER(10,0) DEFAULT NULL, > "y" NUMBER(10,0) DEFAULT NULL >); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2723) Oracle connector not working with lowercase columns
David Robson created SQOOP-2723:
---
Summary: Oracle connector not working with lowercase columns
Key: SQOOP-2723
URL: https://issues.apache.org/jira/browse/SQOOP-2723
Project: Sqoop
Issue Type: Bug
Components: connectors/oracle
Affects Versions: 1.4.6
Reporter: David Robson
Assignee: David Robson
Fix For: 1.4.7
For example the following table does not work for imports or exports:
CREATE TABLE "TestV500"."Xy"
("a" NVARCHAR2(255) DEFAULT NULL,
"x" NUMBER(10,0) DEFAULT NULL,
"y" NUMBER(10,0) DEFAULT NULL
);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-2711) Exit command no longer works in sqoop2-shell
David Robson created SQOOP-2711: --- Summary: Exit command no longer works in sqoop2-shell Key: SQOOP-2711 URL: https://issues.apache.org/jira/browse/SQOOP-2711 Project: Sqoop Issue Type: Bug Components: sqoop2-shell Affects Versions: 1.99.7 Reporter: David Robson A recent change seems to have broken the exit command. Ctrl-D still works as normal. sqoop:000> exit Unknown property: exit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2699) Improve Oracle parameters
David Robson created SQOOP-2699: --- Summary: Improve Oracle parameters Key: SQOOP-2699 URL: https://issues.apache.org/jira/browse/SQOOP-2699 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.99.7 Reporter: David Robson Fix For: 1.99.7 Some of the Oracle parameters need to be improved to use Lists, Enums etc. Can also improve the names of them. Also need to add the ability to filter the columns on the Oracle table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-2699) Improve Oracle parameters
[ https://issues.apache.org/jira/browse/SQOOP-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-2699: Attachment: SQOOP-2699.patch > Improve Oracle parameters > - > > Key: SQOOP-2699 > URL: https://issues.apache.org/jira/browse/SQOOP-2699 > Project: Sqoop > Issue Type: Bug > Components: connectors/oracle >Affects Versions: 1.99.7 >Reporter: David Robson > Fix For: 1.99.7 > > Attachments: SQOOP-2699.patch > > > Some of the Oracle parameters need to be improved to use Lists, Enums etc. > Can also improve the names of them. > Also need to add the ability to filter the columns on the Oracle table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-2595) Add Oracle connector to Sqoop 2
[ https://issues.apache.org/jira/browse/SQOOP-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-2595: Attachment: SQOOP-2595.patch > Add Oracle connector to Sqoop 2 > --- > > Key: SQOOP-2595 > URL: https://issues.apache.org/jira/browse/SQOOP-2595 > Project: Sqoop > Issue Type: Improvement > Components: connectors/oracle >Affects Versions: 1.99.7 >Reporter: David Robson >Assignee: David Robson > Attachments: SQOOP-2595.patch > > > Migrate the code for the Oracle direct connector into Sqoop 2. > Initially this will just be the code from the direct connector in Sqoop 1, > but eventually we need to migrate the code from the other Oracle connector as > well to cover all the scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-2595) Add Oracle connector to Sqoop 2
[ https://issues.apache.org/jira/browse/SQOOP-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-2595: Attachment: SQOOP-2595.patch > Add Oracle connector to Sqoop 2 > --- > > Key: SQOOP-2595 > URL: https://issues.apache.org/jira/browse/SQOOP-2595 > Project: Sqoop > Issue Type: Improvement > Components: connectors/oracle >Affects Versions: 1.99.7 >Reporter: David Robson >Assignee: David Robson > Attachments: SQOOP-2595.patch, SQOOP-2595.patch > > > Migrate the code for the Oracle direct connector into Sqoop 2. > Initially this will just be the code from the direct connector in Sqoop 1, > but eventually we need to migrate the code from the other Oracle connector as > well to cover all the scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2595) Add Oracle connector to Sqoop 2
David Robson created SQOOP-2595: --- Summary: Add Oracle connector to Sqoop 2 Key: SQOOP-2595 URL: https://issues.apache.org/jira/browse/SQOOP-2595 Project: Sqoop Issue Type: Improvement Components: connectors/oracle Affects Versions: 1.99.7 Reporter: David Robson Assignee: David Robson Migrate the code for the Oracle direct connector into Sqoop 2. Initially this will just be the code from the direct connector in Sqoop 1, but eventually we need to migrate the code from the other Oracle connector as well to cover all the scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2464) Initializer object is not reused when calling getSchema
David Robson created SQOOP-2464: --- Summary: Initializer object is not reused when calling getSchema Key: SQOOP-2464 URL: https://issues.apache.org/jira/browse/SQOOP-2464 Project: Sqoop Issue Type: Bug Affects Versions: 1.99.6 Reporter: David Robson In JobManager there is two methods which are called one after the other - initializeConnector and getSchemaForConnector. Both these methods do the same thing as the first step - create a new instance of the initializer class. If the same instance of the initializer was shared it means the class could keep resources open (such as a connection to the database) and not have to re-establish the connection. This might mean a close method needs to be added to the initializers as otherwise the getSchema would need to close any resources opened in the initialize call - which might seem a bit confusing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2465) Initializer and Destroyer should know how many executors will run
David Robson created SQOOP-2465: --- Summary: Initializer and Destroyer should know how many executors will run Key: SQOOP-2465 URL: https://issues.apache.org/jira/browse/SQOOP-2465 Project: Sqoop Issue Type: Bug Affects Versions: 1.99.6 Reporter: David Robson Looking at a job to load data into Oracle as an example - depending on the way the user wants to load data, we may be loading data into temporary tables. For maximum performance we need to create a separate temporary table for each loader - so when the initializer is running we need to know how many loaders will run so we can create these temporary tables. Again when the destroyer is run we will need to drop these temporary tables - so it will need to know as well. Another example where we need to know this in the initializer - Oracle databases may be real application clusters where there is multiple instances across multiple machines. For both FROM and TO jobs we spread the load across these instances during the initialization phase - so we need to know how many loaders / extractors will run. In the case of a FROM job we could do this in the partition phase - but there is no way to achieve this for a TO job. It seems we could either add the information into the initialize phase - or add a new partition phase on the TO side that is called after the partition phase on the FROM side. It could take the details of the partitioned output and match it up to the other side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-2466) Loader needs to be able to uniquely identify itself
David Robson created SQOOP-2466: --- Summary: Loader needs to be able to uniquely identify itself Key: SQOOP-2466 URL: https://issues.apache.org/jira/browse/SQOOP-2466 Project: Sqoop Issue Type: Bug Affects Versions: 1.99.6 Reporter: David Robson Following on from SQOOP-2465 - once the balancing of loaders and instances has been determined, and temporary tables have been created - the loader needs to know which instance it will connect to and which temporary table to use. Currently all loaders are called with the same parameters - so there's no way to distinguish them from each other. The extractor can use the partition input to distinguish itself but there is no such equivalent for loaders. Either the loader needs some sort of id, or an equivalent of the partition input so it can work out which temporary table to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-1764) Numeric Overflow when getting extent map
David Robson created SQOOP-1764: --- Summary: Numeric Overflow when getting extent map Key: SQOOP-1764 URL: https://issues.apache.org/jira/browse/SQOOP-1764 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson User is receiving the following error - looks like the block ID can be bigger than an int: 14/11/06 14:17:58 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.sql.SQLException: Numeric Overflow at com.quest.oraoop.OraOopDataDrivenDBInputFormat.getSplits(OraOopDataDrivenDBInputFormat.java:120) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:498) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:515) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:399) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313) at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:198) at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:171) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:268) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665) at com.quest.oraoop.OraOopConnManager.importTable(OraOopConnManager.java:260) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.sql.SQLException: Numeric Overflow at oracle.jdbc.driver.NumberCommonAccessor.throwOverflow(NumberCommonAccessor.java:4170) at oracle.jdbc.driver.NumberCommonAccessor.getInt(NumberCommonAccessor.java:119) at oracle.jdbc.driver.GeneratedStatement.getInt(GeneratedStatement.java:217) at oracle.jdbc.driver.GeneratedScrollableResultSet.getInt(GeneratedScrollableResultSet.java:522) at oracle.jdbc.driver.GeneratedResultSet.getInt(GeneratedResultSet.java:1350) at com.quest.oraoop.OraOopOracleQueries.getOracleDataChunksExtent(OraOopOracleQueries.java:271) at com.quest.oraoop.OraOopDataDrivenDBInputFormat.getSplits(OraOopDataDrivenDBInputFormat.java:74) ... 23 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1764) Numeric Overflow when getting extent map
[ https://issues.apache.org/jira/browse/SQOOP-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1764: Attachment: SQOOP-1764.patch Numeric Overflow when getting extent map Key: SQOOP-1764 URL: https://issues.apache.org/jira/browse/SQOOP-1764 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Attachments: SQOOP-1764.patch User is receiving the following error - looks like the block ID can be bigger than an int: 14/11/06 14:17:58 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.sql.SQLException: Numeric Overflow at com.quest.oraoop.OraOopDataDrivenDBInputFormat.getSplits(OraOopDataDrivenDBInputFormat.java:120) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:498) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:515) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:399) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313) at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:198) at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:171) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:268) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665) at com.quest.oraoop.OraOopConnManager.importTable(OraOopConnManager.java:260) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.sql.SQLException: Numeric Overflow at oracle.jdbc.driver.NumberCommonAccessor.throwOverflow(NumberCommonAccessor.java:4170) at oracle.jdbc.driver.NumberCommonAccessor.getInt(NumberCommonAccessor.java:119) at oracle.jdbc.driver.GeneratedStatement.getInt(GeneratedStatement.java:217) at oracle.jdbc.driver.GeneratedScrollableResultSet.getInt(GeneratedScrollableResultSet.java:522) at oracle.jdbc.driver.GeneratedResultSet.getInt(GeneratedResultSet.java:1350) at com.quest.oraoop.OraOopOracleQueries.getOracleDataChunksExtent(OraOopOracleQueries.java:271) at com.quest.oraoop.OraOopDataDrivenDBInputFormat.getSplits(OraOopDataDrivenDBInputFormat.java:74) ... 23 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-1663) OraOop test cases are not logging any output
[ https://issues.apache.org/jira/browse/SQOOP-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14197093#comment-14197093 ] David Robson commented on SQOOP-1663: - This was just so we didn't see the entire log for every build - it only outputs the log if there was a problem. In our case it was going to the console - but it's a bit different as Sqoop is already putting them into a log file - so you could probably just remove the removeAllAppenders call and the System.out call and this should mean it always gets logged. OraOop test cases are not logging any output Key: SQOOP-1663 URL: https://issues.apache.org/jira/browse/SQOOP-1663 Project: Sqoop Issue Type: Bug Affects Versions: 1.4.5 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 1.4.6 Output log from OraOop test cases is always suspiciously empty on my box. It seems that the class OraOopTestCase is doing [some dark magic|https://github.com/apache/sqoop/blob/trunk/src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java#L188] with appenders that results in the empty log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1632) Add support for index organized tables to direct connector
[ https://issues.apache.org/jira/browse/SQOOP-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1632: Attachment: SQOOP-1632.patch Add support for index organized tables to direct connector -- Key: SQOOP-1632 URL: https://issues.apache.org/jira/browse/SQOOP-1632 Project: Sqoop Issue Type: Improvement Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Attachments: SQOOP-1632.patch, SQOOP-1632.patch If an index organized table is partitioned - we can use the new chunk by partition functionality on it - so we should add support for this. IOTs that are not partitioned will still not be supported by the direct connector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-1632) Add support for index organized tables to direct connector
David Robson created SQOOP-1632: --- Summary: Add support for index organized tables to direct connector Key: SQOOP-1632 URL: https://issues.apache.org/jira/browse/SQOOP-1632 Project: Sqoop Issue Type: Improvement Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson If an index organized table is partitioned - we can use the new chunk by partition functionality on it - so we should add support for this. IOTs that are not partitioned will still not be supported by the direct connector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1433) Oracle direct connector fails with Oracle 12c JDBC driver
[ https://issues.apache.org/jira/browse/SQOOP-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1433: Attachment: SQOOP-1433.patch Oracle direct connector fails with Oracle 12c JDBC driver - Key: SQOOP-1433 URL: https://issues.apache.org/jira/browse/SQOOP-1433 Project: Sqoop Issue Type: Bug Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Fix For: 1.4.6 Attachments: SQOOP-1433.patch If you issue a commit statement while auto commit is turned on with the Oracle 12c JDBC driver it throws an error where the 11g drivers did not. This causes Sqoop to fail so this needs to be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-1486) Oracle direct connector should use Sqoop merge syntax
David Robson created SQOOP-1486: --- Summary: Oracle direct connector should use Sqoop merge syntax Key: SQOOP-1486 URL: https://issues.apache.org/jira/browse/SQOOP-1486 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson From [~gwenshap] when running the following sqoop command with OraOop installed: sqoop export --table mytable --update-key COL1, COL2, COL3 --export-dir /user/hive/warehouse/mytable --connect jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 1.1.1.1)(PORT = 1521)))(SDU = 5844)(CONNECT_DATA = (ORACLE_SID = oracle))) --username scott --password tiger The job runs succesfully and reports: INFO mapreduce.ExportJobBase: Exported 54 records. However not rows actually appears in the table. It appears that Oraoop loads the data into staging tables, but not into the user table. When using the Oraoop specific syntax, the export succeeds and we see data in the user table: sqoop export -Doraoop.export.merge=true -Doraoop.update.key.extra.columns=COL2 --table mytable --export-dir /user/hive/warehouse/mytable --connect jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 1.1.1.1)(PORT = 1521)))(SDU = 5844)(CONNECT_DATA = (ORACLE_SID = oracle))) --username scott --password tiger --input-fields-terminated-by ',' --update-key COL1 Expected behavior: Either make sure the usual Sqoop syntax works with Oraoop, or issue a suitable error message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SQOOP-1433) Oracle direct connector fails with Oracle 12c JDBC driver
[ https://issues.apache.org/jira/browse/SQOOP-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096384#comment-14096384 ] David Robson commented on SQOOP-1433: - Yes looks like the same issue! We can just do the same thing for the direct connector. Oracle direct connector fails with Oracle 12c JDBC driver - Key: SQOOP-1433 URL: https://issues.apache.org/jira/browse/SQOOP-1433 Project: Sqoop Issue Type: Bug Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Fix For: 1.4.6 If you issue a commit statement while auto commit is turned on with the Oracle 12c JDBC driver it throws an error where the 11g drivers did not. This causes Sqoop to fail so this needs to be fixed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SQOOP-1433) Oracle direct connector fails with Oracle 12c JDBC driver
David Robson created SQOOP-1433: --- Summary: Oracle direct connector fails with Oracle 12c JDBC driver Key: SQOOP-1433 URL: https://issues.apache.org/jira/browse/SQOOP-1433 Project: Sqoop Issue Type: Bug Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Fix For: 1.4.6 If you issue a commit statement while auto commit is turned on with the Oracle 12c JDBC driver it throws an error where the 11g drivers did not. This causes Sqoop to fail so this needs to be fixed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SQOOP-1422) Integration tests for Oracle connector fail as not using direct option
David Robson created SQOOP-1422: --- Summary: Integration tests for Oracle connector fail as not using direct option Key: SQOOP-1422 URL: https://issues.apache.org/jira/browse/SQOOP-1422 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Priority: Minor Made the Oracle connector require the direct flag to activate, but didn't update the integration tests with this. So they fail now - need to add the direct flag to integration tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SQOOP-1422) Integration tests for Oracle connector fail as not using direct option
[ https://issues.apache.org/jira/browse/SQOOP-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1422: Attachment: SQOOP-1422.patch Integration tests for Oracle connector fail as not using direct option -- Key: SQOOP-1422 URL: https://issues.apache.org/jira/browse/SQOOP-1422 Project: Sqoop Issue Type: Bug Components: connectors/oracle Affects Versions: 1.4.5 Reporter: David Robson Assignee: David Robson Priority: Minor Attachments: SQOOP-1422.patch Made the Oracle connector require the direct flag to activate, but didn't update the integration tests with this. So they fail now - need to add the direct flag to integration tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SQOOP-1388) One of Export scenario fails with an error : java.sql.SQLException: Unable to perform an exchange subpartition operation for the table ...
[ https://issues.apache.org/jira/browse/SQOOP-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068182#comment-14068182 ] David Robson commented on SQOOP-1388: - This is caused because the requested number of splits is 4, but CombineFileInputFormat is generating 5 splits. OraOop is assuming the requested number of splits will be the actual number of splits. Need to confirm if this is meant to be the case or not. One of Export scenario fails with an error : java.sql.SQLException: Unable to perform an exchange subpartition operation for the table ... - Key: SQOOP-1388 URL: https://issues.apache.org/jira/browse/SQOOP-1388 Project: Sqoop Issue Type: Bug Environment: sqoop-1.4.5-SNAPSHOT.bin__hadoop-2.0.4-alpha.tar.gz on CDH4 (melhadoop 01). Reporter: Vidya Iyengar Assignee: David Robson Ran Export Scenario 13 from the Guy Ley Mar document on SASDB01. These are the steps I followed for CASE 13. drop table junk; drop table junk13; create table junk as select * from dba_objects; sqoop import --direct --connect jdbc:oracle:thin:@melsasdb01:1521:sasdb01 --username oraoop --password oraoop --table junk --where OBJECT_ID100 --target-dir junk_a sqoop import --direct --connect jdbc:oracle:thin:@melsasdb01:1521:sasdb01 --username oraoop --password oraoop --table junk --where OBJECT_ID= 100 and OBJECT_ID 200 --target-dir junk_b sqoop export -D oraoop.drop.table=true -Doraoop.template.table=oraoop.junk -Doraoop.nologging=true -Doraoop.partitioned=true -Doraoop.export.merge=false --direct --connect jdbc:oracle:thin:@melsasdb01:1521:sasdb01 --username oraoop --password oraoop --table junk13 --export-dir junk_a sqoop export -Doraoop.export.merge=false --direct --connect jdbc:oracle:thin:@melsasdb01:1521:sasdb01 --username oraoop --password oraoop --table junk13 --export-dir junk_b Note: Failed when i ran the first export. On the Hadoop file system, only one file contains data remaining 3 files are with size 0. Removing empty files export worked ok. Imported folders are available junk_a_bug , junk_b_bug for debugging. Attached file contains the output error. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SQOOP-1344) Add documentation for Oracle connector
[ https://issues.apache.org/jira/browse/SQOOP-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1344: Attachment: SQOOP-1344.patch Add documentation for Oracle connector -- Key: SQOOP-1344 URL: https://issues.apache.org/jira/browse/SQOOP-1344 Project: Sqoop Issue Type: Task Components: connectors/oracle Reporter: David Robson Assignee: David Robson Fix For: 1.4.5 Attachments: SQOOP-1344.patch Now the connector is included in Sqoop we need to add any relevant details from the OraOop documentation into the Sqoop documentation. This will be in the connectors section of the documentation as discussed on the mailing list. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SQOOP-732) Fix checkstyle violations on trunk
[ https://issues.apache.org/jira/browse/SQOOP-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058457#comment-14058457 ] David Robson commented on SQOOP-732: Richard - if you run the checkstyle target from ant - (ant checkstyle in the root directory) - the build will generate a report with violations in build/checkstyle-errors.html Currently on trunk there are 235 violations that I can see. Fix checkstyle violations on trunk -- Key: SQOOP-732 URL: https://issues.apache.org/jira/browse/SQOOP-732 Project: Sqoop Issue Type: Task Reporter: Jarek Jarcec Cecho Priority: Trivial Labels: newbie Attachments: patch.diff We have checkstyle violations on trunk that should be fixed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SQOOP-1357) QA testing of Data Connector for Oracle and Hadoop
[ https://issues.apache.org/jira/browse/SQOOP-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-1357: Assignee: Vidya Iyengar QA testing of Data Connector for Oracle and Hadoop -- Key: SQOOP-1357 URL: https://issues.apache.org/jira/browse/SQOOP-1357 Project: Sqoop Issue Type: Sub-task Components: connectors/oracle Reporter: David Robson Assignee: Vidya Iyengar Fix For: 1.4.5 Original Estimate: 48h Remaining Estimate: 48h As part of donating the data connector - Dell would like to perform QA before it is released. This should be started on the 16th July. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SQOOP-1357) QA testing of Data Connector for Oracle and Hadoop
David Robson created SQOOP-1357: --- Summary: QA testing of Data Connector for Oracle and Hadoop Key: SQOOP-1357 URL: https://issues.apache.org/jira/browse/SQOOP-1357 Project: Sqoop Issue Type: Sub-task Components: connectors/oracle Reporter: David Robson Fix For: 1.4.5 As part of donating the data connector - Dell would like to perform QA before it is released. This should be started on the 16th July. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SQOOP-1287) Add high performance Oracle connector into Sqoop
[ https://issues.apache.org/jira/browse/SQOOP-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047063#comment-14047063 ] David Robson commented on SQOOP-1287: - Hey Venkat - I checked them they are the same file - what problems do you mean? Yes don't really need the extra option - I left it there just in case someone tried to set it to disable OraOop out of habit... Add high performance Oracle connector into Sqoop Key: SQOOP-1287 URL: https://issues.apache.org/jira/browse/SQOOP-1287 Project: Sqoop Issue Type: New Feature Components: connectors/oracle Reporter: David Robson Assignee: David Robson Fix For: 1.4.5 Attachments: SQOOP-1287.patch Dell Software is contributing an Oracle connector for the Sqoop project. We had a discussion about this on the mailing list (http://markmail.org/thread/42tlyjzg5ip245jq), and also a vote (http://markmail.org/thread/wjxpv6zxeo2ioaqj). The results of the vote are here http://markmail.org/message/366aqe6rqy4yji6j -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SQOOP-1287) Add high performance Oracle connector into Sqoop
[ https://issues.apache.org/jira/browse/SQOOP-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047357#comment-14047357 ] David Robson commented on SQOOP-1287: - Thanks Venkat - looks good. Yeah I think there must have been some Windows line endings in there somehow - I'm not sure how they got there as have only used a linux machine for this! Anyway sounds like you sorted it out. Add high performance Oracle connector into Sqoop Key: SQOOP-1287 URL: https://issues.apache.org/jira/browse/SQOOP-1287 Project: Sqoop Issue Type: New Feature Components: connectors/oracle Reporter: David Robson Assignee: David Robson Fix For: 1.4.5 Attachments: SQOOP-1287.patch Dell Software is contributing an Oracle connector for the Sqoop project. We had a discussion about this on the mailing list (http://markmail.org/thread/42tlyjzg5ip245jq), and also a vote (http://markmail.org/thread/wjxpv6zxeo2ioaqj). The results of the vote are here http://markmail.org/message/366aqe6rqy4yji6j -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SQOOP-1344) Add documentation for Oracle connector
David Robson created SQOOP-1344: --- Summary: Add documentation for Oracle connector Key: SQOOP-1344 URL: https://issues.apache.org/jira/browse/SQOOP-1344 Project: Sqoop Issue Type: Task Components: connectors/oracle Reporter: David Robson Assignee: David Robson Fix For: 1.4.5 Now the connector is included in Sqoop we need to add any relevant details from the OraOop documentation into the Sqoop documentation. This will be in the connectors section of the documentation as discussed on the mailing list. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (SQOOP-1345) Merge the two Oracle connectors into one
David Robson created SQOOP-1345: --- Summary: Merge the two Oracle connectors into one Key: SQOOP-1345 URL: https://issues.apache.org/jira/browse/SQOOP-1345 Project: Sqoop Issue Type: Task Components: connectors/oracle Reporter: David Robson The donated Oracle connector is currently sitting as a --direct connector, and the moment. We should just have one Oracle connector - taking the bits that each connector does and combining it into one. This is a good opportunity to refactor some of it - for example moving some of the classes to different packages, improving some methods that are too long etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (SQOOP-862) Hbase import fails if there is a row where all columns are null
[ https://issues.apache.org/jira/browse/SQOOP-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-862: --- Attachment: SQOOP-862.patch Hbase import fails if there is a row where all columns are null --- Key: SQOOP-862 URL: https://issues.apache.org/jira/browse/SQOOP-862 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-862.patch, SQOOP-862.patch If you try to import a table where any of the rows contain all null values (except for the primary key), the import fails. For example create the following table in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 123); INSERT INTO employee values(2, null); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The Import fails with: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400) Caused by: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:950) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:766) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752) at org.apache.sqoop.hbase.HBasePutProcessor.accept(HBasePutProcessor.java:127) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:128) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:1) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:598) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:38) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-862) Hbase import fails if there is a row where all columns are null
[ https://issues.apache.org/jira/browse/SQOOP-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573966#comment-13573966 ] David Robson commented on SQOOP-862: Updated patch with documentation changes. Hbase import fails if there is a row where all columns are null --- Key: SQOOP-862 URL: https://issues.apache.org/jira/browse/SQOOP-862 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-862.patch, SQOOP-862.patch If you try to import a table where any of the rows contain all null values (except for the primary key), the import fails. For example create the following table in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 123); INSERT INTO employee values(2, null); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The Import fails with: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400) Caused by: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:950) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:766) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752) at org.apache.sqoop.hbase.HBasePutProcessor.accept(HBasePutProcessor.java:127) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:128) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:1) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:598) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:38) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (SQOOP-870) Allow Sqoop to import row key column into HBase
David Robson created SQOOP-870: -- Summary: Allow Sqoop to import row key column into HBase Key: SQOOP-870 URL: https://issues.apache.org/jira/browse/SQOOP-870 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Currently when importing data into HBase, the row key column is not added to the column family as it is already the key. I would like to be able to tell Sqoop to add the row key as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-870) Allow Sqoop to import row key column into HBase
[ https://issues.apache.org/jira/browse/SQOOP-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-870: --- Attachment: SQOOP-870.patch Allow Sqoop to import row key column into HBase --- Key: SQOOP-870 URL: https://issues.apache.org/jira/browse/SQOOP-870 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-870.patch Currently when importing data into HBase, the row key column is not added to the column family as it is already the key. I would like to be able to tell Sqoop to add the row key as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (SQOOP-862) Hbase import fails if there is a row where all columns are null
David Robson created SQOOP-862: -- Summary: Hbase import fails if there is a row where all columns are null Key: SQOOP-862 URL: https://issues.apache.org/jira/browse/SQOOP-862 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson If you try to import a table where any of the rows contain all null values (except for the primary key), the import fails. For example create the following table in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 123); INSERT INTO employee values(2, null); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The Import fails with: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400) Caused by: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:950) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:766) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752) at org.apache.sqoop.hbase.HBasePutProcessor.accept(HBasePutProcessor.java:127) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:128) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:1) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:598) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:38) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-862) Hbase import fails if there is a row where all columns are null
[ https://issues.apache.org/jira/browse/SQOOP-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-862: --- Attachment: SQOOP-862.patch Hbase import fails if there is a row where all columns are null --- Key: SQOOP-862 URL: https://issues.apache.org/jira/browse/SQOOP-862 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-862.patch If you try to import a table where any of the rows contain all null values (except for the primary key), the import fails. For example create the following table in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 123); INSERT INTO employee values(2, null); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The Import fails with: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400) Caused by: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:950) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:766) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752) at org.apache.sqoop.hbase.HBasePutProcessor.accept(HBasePutProcessor.java:127) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:128) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:1) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:598) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:38) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-862) Hbase import fails if there is a row where all columns are null
[ https://issues.apache.org/jira/browse/SQOOP-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571013#comment-13571013 ] David Robson commented on SQOOP-862: Added defensive code to skip the HBase put and warn the user if it has no columns and would therefore fail. Added a new unit test for this issue. Hbase import fails if there is a row where all columns are null --- Key: SQOOP-862 URL: https://issues.apache.org/jira/browse/SQOOP-862 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-862.patch If you try to import a table where any of the rows contain all null values (except for the primary key), the import fails. For example create the following table in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 123); INSERT INTO employee values(2, null); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The Import fails with: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400) Caused by: java.lang.IllegalArgumentException: No columns to insert at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:950) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:766) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752) at org.apache.sqoop.hbase.HBasePutProcessor.accept(HBasePutProcessor.java:127) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:128) at org.apache.sqoop.mapreduce.DelegatingOutputFormat$DelegatingRecordWriter.write(DelegatingOutputFormat.java:1) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:598) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:38) at org.apache.sqoop.mapreduce.HBaseImportMapper.map(HBaseImportMapper.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-830) HBase import formatting BigDecimal inconsistently
[ https://issues.apache.org/jira/browse/SQOOP-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-830: --- Attachment: SQOOP-830.patch HBase import formatting BigDecimal inconsistently - Key: SQOOP-830 URL: https://issues.apache.org/jira/browse/SQOOP-830 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-830.patch, SQOOP-830.patch When importing into HBase the toString() method is called on every field via the ToStringPutTransformer class. When the field is mapped as a BigDecimal - as it is with number fields in Oracle - this results in inconsistent formats in HBase. For example - create the following in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 0.01); INSERT INTO employee values(2, 0.001); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The value for row 1 is 0.01 while row 2 is 1E-7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-830) HBase import formatting BigDecimal inconsistently
[ https://issues.apache.org/jira/browse/SQOOP-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-830: --- Attachment: SQOOP-830.patch HBase import formatting BigDecimal inconsistently - Key: SQOOP-830 URL: https://issues.apache.org/jira/browse/SQOOP-830 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Attachments: SQOOP-830.patch When importing into HBase the toString() method is called on every field via the ToStringPutTransformer class. When the field is mapped as a BigDecimal - as it is with number fields in Oracle - this results in inconsistent formats in HBase. For example - create the following in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 0.01); INSERT INTO employee values(2, 0.001); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The value for row 1 is 0.01 while row 2 is 1E-7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-830) HBase import formatting BigDecimal inconsistently
[ https://issues.apache.org/jira/browse/SQOOP-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson reassigned SQOOP-830: -- Assignee: David Robson HBase import formatting BigDecimal inconsistently - Key: SQOOP-830 URL: https://issues.apache.org/jira/browse/SQOOP-830 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson When importing into HBase the toString() method is called on every field via the ToStringPutTransformer class. When the field is mapped as a BigDecimal - as it is with number fields in Oracle - this results in inconsistent formats in HBase. For example - create the following in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 0.01); INSERT INTO employee values(2, 0.001); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The value for row 1 is 0.01 while row 2 is 1E-7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (SQOOP-830) HBase import formatting BigDecimal inconsistently
David Robson created SQOOP-830: -- Summary: HBase import formatting BigDecimal inconsistently Key: SQOOP-830 URL: https://issues.apache.org/jira/browse/SQOOP-830 Project: Sqoop Issue Type: Bug Reporter: David Robson When importing into HBase the toString() method is called on every field via the ToStringPutTransformer class. When the field is mapped as a BigDecimal - as it is with number fields in Oracle - this results in inconsistent formats in HBase. For example - create the following in Oracle: CREATE TABLE employee(id number primary key, test_number number); INSERT INTO employee values(1, 0.01); INSERT INTO employee values(2, 0.001); COMMIT; Then run an import: sqoop import --connect jdbc:oracle:thin:@//HOSTNAME/SERVICE --username USERNAME --table EMPLOYEE --password PASSWORD --hbase-table EMPLOYEE --column-family tst --hbase-create-table The value for row 1 is 0.01 while row 2 is 1E-7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson updated SQOOP-382: --- Attachment: SQOOP-382.patch Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Fix For: 1.4.3 Attachments: SQOOP-382.patch Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504273#comment-13504273 ] David Robson commented on SQOOP-382: Thanks Jarcec - for anyone using the Oracle Connector I have fixed it there as well: https://questmos.jira.com/browse/ORAOOP-18 Jan - The Microsoft connector may need to be updated as well - you might want to point them at this issue so they can fix it as well. Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Fix For: 1.4.3 Attachments: SQOOP-382.patch Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Robson reassigned SQOOP-382: -- Assignee: David Robson (was: Jarek Jarcec Cecho) Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Fix For: 1.4.3 Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503621#comment-13503621 ] David Robson commented on SQOOP-382: Added review request https://reviews.apache.org/r/8221/ Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: David Robson Fix For: 1.4.3 Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502968#comment-13502968 ] David Robson commented on SQOOP-382: It looks like you need to add the jar to the classpath - eg java TestAzure -cp ./:./sqljdbc4.jar You also probably need to add the connection string to your Java code to actually create the connection. As for your problem - I asked a colleague who has used Sqoop on Azure before and he gave me the following command line he used which worked successfully: sqoop import --connect jdbc:sqlserver://YOUR_HOST.database.windows.net:1433;database=YOUR_DATABASE;user=YOUR_USER@YOUR_HOST;password=YOUR_PASSWORD --table YOUR_TABLE --target-dir YOUR_DIRECTORY --split-by YOUR_SPLIT_COLUMN Could you try this command - so get rid of the connection parameters completely - and specify the password in the URL just to see if it works? From my experience the connection parameters using the connection-param-file do not work (hence this bug) - so probably best to eliminate them all together. Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502978#comment-13502978 ] David Robson commented on SQOOP-382: It looks to me like your problem might be with the Microsoft connector itself - I am not sure where to get support for that as I am not familiar with it - but seems like a bug they should fix. Anyway - in regards to SQOOP-382 the connection parameters don't work - should this be fixed in 1.x branch or should we leave it to 2.x? If it's not going to be fixed in 1.x should we at least update the documentation warning people is does not work? Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-382) Connection parameters should be used on the mapper
[ https://issues.apache.org/jira/browse/SQOOP-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502989#comment-13502989 ] David Robson commented on SQOOP-382: I actually looked into this a fair bit when I raised it - I think the easiest way is to just make it a Configuration parameter - then you can use the standard Hadoop Configuration object to get the array of connection parameters. Part of the problem with this for Sqoop 1.x is each connector has to implement it - so while it can be fixed in Sqoop - it won't work in say the Microsoft connector or the Quest one. Of course if it's fixed I can fix the Quest connector... Let me know if this feature is desirable - seeing as I'd have to fix OraOop anyway I could fix it in Sqoop at the same time and submit a patch. Connection parameters should be used on the mapper -- Key: SQOOP-382 URL: https://issues.apache.org/jira/browse/SQOOP-382 Project: Sqoop Issue Type: Bug Reporter: David Robson Assignee: Jarek Jarcec Cecho Fix For: 1.4.3 Currently you can specify connection parameters using --connection-param-file properties-file. This applies the connection parameters to the connection when generating the Sqoop code - but the parameters are not passed down to the mapper. Instead of specifying a parameters file couldn't we have a comma seperated list that could be specified on the command line or in sqoop-site.xml - that way it would be easier to override the settings per job, and they would be passed down to the mappers. It would then be simple to modify DBConfiguration.getConnection to read these. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
