[jira] [Resolved] (SQOOP-3211) Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704)
[
https://issues.apache.org/jira/browse/SQOOP-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ying Cao resolved SQOOP-3211.
-
Resolution: Not A Problem
Assignee: Ying Cao
this is a typical error of DB2 is not case-sensitive database, for the
lower-case table name please use \"\" to make table name
> Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table
> fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704)
> ---
>
> Key: SQOOP-3211
> URL: https://issues.apache.org/jira/browse/SQOOP-3211
> Project: Sqoop
> Issue Type: Bug
> Environment: $ sqoop version
> 17/07/17 14:22:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.1
>Reporter: Markus Kemper
>Assignee: Ying Cao
>
> Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table
> fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704), see test case
> below.
> *Test Case*
> {noformat}
> #
> # STEP 01 - Create Table and Data
> #
> export MYCONN=jdbc:db2://host.domain.com:5/SQOOP
> export MYUSER=sqoop;
> export MYPSWD=sqoop;
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "drop table t1_default"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "create table t1_default (c1_default int)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "insert into t1_default values (1)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select c1_default from t1_default"
> ---
> | C1_DEFAULT |
> ---
> | 1 |
> ---
> #
> # STEP 02 - Import Data (baseline) using (--as-textfile, --as-avrodatafile,
> --as-parquetfile)
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-textfile
> hdfs dfs -cat /user/root/t1_default/part*
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-avrodatafile
> avro-tools tojson --pretty
> 'hdfs://host.domain.com/user/root/t1_default/part-m-0.avro'
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-parquetfile
> hdfs dfs -ls /user/root/t1_default/*.parquet
> parquet-tools cat --json
> 'hdfs://host.domain.com/user/root/t1_default/c2a7687c-b2bd-40d6-8959-d8ce9c240ae6.parquet'
> Output:
> 17/07/17 14:00:41 INFO mapreduce.ImportJobBase: Transferred 2 bytes in
> 30.2579 seconds (0.0661 bytes/sec)
> 17/07/17 14:00:41 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> 1
> ~
> 17/07/17 14:02:05 INFO mapreduce.ImportJobBase: Transferred 263 bytes in
> 22.8116 seconds (11.5292 bytes/sec)
> 17/07/17 14:02:05 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> {
> "C1_DEFAULT" : {
> "int" : 1
> }
> }
> ~
> 17/07/17 14:06:29 INFO mapreduce.ImportJobBase: Transferred 1.2578 KB in
> 29.3071 seconds (43.9484 bytes/sec)
> 17/07/17 14:06:29 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> -rw-r--r-- 3 root root449 2017-07-17 14:06
> /user/root/t1_default/c2a7687c-b2bd-40d6-8959-d8ce9c240ae6.parquet
> ~
> {"C1_DEFAULT":1}
> #
> # STEP 03 - Create Table and Data with case-sensitive DB2 table name
> #
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "drop table \"t1_lower\""
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "create table \"t1_lower\" (\"c1_lower\" int)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "insert into \"t1_lower\" values (1)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select \"c1_lower\" from \"t1_lower\""
> sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD |
> egrep -i "t1"
> Output:
> ---
> | c1_lower|
> ---
> | 1 |
[jira] [Commented] (SQOOP-3211) Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704)
[
https://issues.apache.org/jira/browse/SQOOP-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102938#comment-16102938
]
Ying Cao commented on SQOOP-3211:
-
The return DB2 error of SQLCODE=-204 is a typical error for "can not find
table".
>From details error:
_17/07/17 14:14:47 INFO manager.SqlManager: Executing SQL statement: *SELECT
t.* FROM t1_lower AS t WHERE 1=0*
17/07/17 14:14:47 ERROR manager.SqlManager: Error executing statement:
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204,
SQLSTATE=42704, SQLERRMC=DB2INST1.T1_LOWER, DRIVER=4.16.53
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: *SQLCODE=-204*,
SQLSTATE=42704, SQLERRMC=DB2INST1.T1_LOWER, DRIVER=4.16.53
at com.ibm.db2.jcc.am.fd.a(fd.java:739)_
Sqoop invoke sql "SELECT t.* FROM t1_lower AS t WHERE 1=0" to DB2, but DB2 is
not case-sensitive which mean this statement works on table "T1_LOWER" instead
of "t1_lower", so DB2 returns SQLCODE 204 it works well.
Let's return to you issue, special table name with \"\" to make table name
lower case and mark with "" will help to fix the error.
For example : import a DB2 table "test"
_sqoop export -connect < --username --password --table
\"\"test\"\"_
then it will try to access table "test": _ INFO manager.SqlManager: Executing
SQL statement: SELECT t.* FROM "test" AS t WHERE 1=0_
> Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table
> fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704)
> ---
>
> Key: SQOOP-3211
> URL: https://issues.apache.org/jira/browse/SQOOP-3211
> Project: Sqoop
> Issue Type: Bug
> Environment: $ sqoop version
> 17/07/17 14:22:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.1
>Reporter: Markus Kemper
>
> Sqoop1 (import + --target-dir + --table) using DB2 with case-sensitive-table
> fails with error (DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704), see test case
> below.
> *Test Case*
> {noformat}
> #
> # STEP 01 - Create Table and Data
> #
> export MYCONN=jdbc:db2://host.domain.com:5/SQOOP
> export MYUSER=sqoop;
> export MYPSWD=sqoop;
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "drop table t1_default"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "create table t1_default (c1_default int)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "insert into t1_default values (1)"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select c1_default from t1_default"
> ---
> | C1_DEFAULT |
> ---
> | 1 |
> ---
> #
> # STEP 02 - Import Data (baseline) using (--as-textfile, --as-avrodatafile,
> --as-parquetfile)
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-textfile
> hdfs dfs -cat /user/root/t1_default/part*
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-avrodatafile
> avro-tools tojson --pretty
> 'hdfs://host.domain.com/user/root/t1_default/part-m-0.avro'
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1_default --target-dir /user/root/t1_default --delete-target-dir
> --num-mappers 1 --as-parquetfile
> hdfs dfs -ls /user/root/t1_default/*.parquet
> parquet-tools cat --json
> 'hdfs://host.domain.com/user/root/t1_default/c2a7687c-b2bd-40d6-8959-d8ce9c240ae6.parquet'
> Output:
> 17/07/17 14:00:41 INFO mapreduce.ImportJobBase: Transferred 2 bytes in
> 30.2579 seconds (0.0661 bytes/sec)
> 17/07/17 14:00:41 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> 1
> ~
> 17/07/17 14:02:05 INFO mapreduce.ImportJobBase: Transferred 263 bytes in
> 22.8116 seconds (11.5292 bytes/sec)
> 17/07/17 14:02:05 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> {
> "C1_DEFAULT" : {
> "int" : 1
> }
> }
> ~
> 17/07/17 14:06:29 INFO mapreduce.ImportJobBase: Transferred 1.2578 KB in
> 29.3071 seconds (43.9484 bytes/sec)
> 17/07/17 14:06:29 INFO mapreduce.ImportJobBase: Retr
[jira] [Commented] (SQOOP-3199) Sqoop 1.4.7 release preparation
[ https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057244#comment-16057244 ] Ying Cao commented on SQOOP-3199: - Hi Attila, As discussed in the mail, please include following JIRA in the new release: https://issues.apache.org/jira/browse/SQOOP-1905 Regards, Angela > Sqoop 1.4.7 release preparation > --- > > Key: SQOOP-3199 > URL: https://issues.apache.org/jira/browse/SQOOP-3199 > Project: Sqoop > Issue Type: Task >Reporter: Attila Szabo >Assignee: Attila Szabo > Fix For: no-release > > > Umbrella jira for 1.4.6 release. > For reference, the release wikis are: > https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release > https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
> On 六月 5, 2017, 11:09 a.m., Boglarka Egyed wrote: > > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManual.java > > Lines 76 (patched) > > <https://reviews.apache.org/r/58600/diff/3/?file=1741029#file1741029line76> > > > > Please use ManualTest at the end so that the related ant task for > > manual tests will find it. Hi Bogi, Thanks for your remind, I have update fix, please help review it again. Regards, Angela - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/#review176905 ------- On 六月 5, 2017, 3:01 p.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58600/ > --- > > (Updated 六月 5, 2017, 3:01 p.m.) > > > Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, > Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-1905 > https://issues.apache.org/jira/browse/SQOOP-1905 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1905 : add --schema option for import-all-tables and list-tables > against db2 > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef > src/java/org/apache/sqoop/manager/GenericJdbcManager.java 2113a5f5 > > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58600/diff/4/ > > > Testing > --- > > Mannual UT is passed > > > Thanks, > > Ying Cao > >
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
> On 五月 24, 2017, 3:15 p.m., Boglarka Egyed wrote: > > Hi Ying, > > > > I tried to run 'ant clean test' with your patch but ran into some > > compilation errors. Please find the details below. > > > > Thanks, > > Bogi > > Attila Szabo wrote: > Hey Bogi, > > Thanks for checking, and catching these issues! > > Angela! Please also fix the issues what Bogi raised, and around the > logging you could also consider to do some generalization (as I can see > you're using LoggingUtils and the Logger itself in a kinda mixed way, > depending you your personal preferences it would be nice, to use only one of > these ways for logging). > > Thanks, > Attila Hi Attila, I have fix issues Bogi raised, thanks Bogi detials. The manual test is passed And I have make some updates use loggingUtils to pop-up SQLException, Logger for others. > On 五月 24, 2017, 3:15 p.m., Boglarka Egyed wrote: > > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchema.java > > Lines 73 (patched) > > <https://reviews.apache.org/r/58600/diff/2/?file=1731302#file1731302line73> > > > > Could you please include "ManualTest" at the end of this test name to > > follow the usual naming convention which is also used in the build.xml to > > create the manual test suite? Hi Bogi, Thnaks for your comments Angela - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/#review175936 --- On 六月 5, 2017, 3:01 p.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58600/ > --- > > (Updated 六月 5, 2017, 3:01 p.m.) > > > Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, > Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-1905 > https://issues.apache.org/jira/browse/SQOOP-1905 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1905 : add --schema option for import-all-tables and list-tables > against db2 > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef > src/java/org/apache/sqoop/manager/GenericJdbcManager.java 2113a5f5 > > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58600/diff/4/ > > > Testing > --- > > Mannual UT is passed > > > Thanks, > > Ying Cao > >
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/ --- (Updated 六月 5, 2017, 3:01 p.m.) Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, Attila Szabo, and Szabolcs Vasas. Changes --- Fix base on Bogi's comments Bugs: SQOOP-1905 https://issues.apache.org/jira/browse/SQOOP-1905 Repository: sqoop-trunk Description --- SQOOP-1905 : add --schema option for import-all-tables and list-tables against db2 Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef src/java/org/apache/sqoop/manager/GenericJdbcManager.java 2113a5f5 src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/58600/diff/4/ Changes: https://reviews.apache.org/r/58600/diff/3-4/ Testing --- Mannual UT is passed Thanks, Ying Cao
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
> On 五月 17, 2017, 11:50 a.m., Attila Szabo wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java > > Lines 372-394 (patched) > > <https://reviews.apache.org/r/58600/diff/1/?file=1696010#file1696010line372> > > > > Could you please check if this has been already solved in other > > connectors? > > > > Getting schema option sounds quite general, might have been implemented > > elsewhere in the codebase. > > > > Would make sense to abstract it out, and just call it from different > > places. > > > > Thanks for checking! > > Ying Cao wrote: > Hi Attila > > Yes, this has been already solved in Postgresql. Which file is suitable > to abstrace it out? > > Attila Szabo wrote: > Hey Angela, > > I would recommend GenericJdbcManager class. > > Cheers, > Attila Hi Attila, Thanks for your suggestions. I have abstrace it in GenericJdbcManager class Angela - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/#review175233 --- On 六月 4, 2017, 4:38 p.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58600/ > --- > > (Updated 六月 4, 2017, 4:38 p.m.) > > > Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, > Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-1905 > https://issues.apache.org/jira/browse/SQOOP-1905 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1905 : add --schema option for import-all-tables and list-tables > against db2 > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef > src/java/org/apache/sqoop/manager/GenericJdbcManager.java 2113a5f5 > > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManual.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58600/diff/3/ > > > Testing > --- > > Mannual UT is passed > > > Thanks, > > Ying Cao > >
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/ --- (Updated 六月 4, 2017, 4:38 p.m.) Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, Attila Szabo, and Szabolcs Vasas. Bugs: SQOOP-1905 https://issues.apache.org/jira/browse/SQOOP-1905 Repository: sqoop-trunk Description --- SQOOP-1905 : add --schema option for import-all-tables and list-tables against db2 Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef src/java/org/apache/sqoop/manager/GenericJdbcManager.java 2113a5f5 src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManual.java PRE-CREATION Diff: https://reviews.apache.org/r/58600/diff/3/ Changes: https://reviews.apache.org/r/58600/diff/2-3/ Testing --- Mannual UT is passed Thanks, Ying Cao
Re: SQOOP 1.4.7 Release by end of June
Hi Attila, This is really great, +1 to the idea ! 2017-05-30 23:40 GMT+08:00 Szabolcs Vasas <va...@cloudera.com>: > Hi Attila, > > Thank you for taking the initiative, big +1 to the idea! > > Regards, > Szabolcs > > On Tue, May 30, 2017 at 1:40 PM, Anna Szonyi <szo...@cloudera.com> wrote: > > > Hi Attila, > > > > This would be really great! Please let me know if I can help with > anything! > > > > Thanks, > > Anna > > > > On Thu, May 25, 2017 at 6:20 PM, Attila Szabó <mau...@apache.org> wrote: > > > > > Hello everyone, > > > > > > In the past few months we've mentioned it a few times that it would > make > > > sense to create a new release of Sqoop from the current trunk version > > > (1.4.7). > > > > > > IMHO this is the right time to move forward with this topic. > > > First of all because the latest release is 2 years old (1.4.6 was > > released > > > in 2015 May), and it would make a great impact for the end users to > > deliver > > > all of the changes we as a community have made since. > > > As a second argument, recently we were able to onboard quite a few new > > > contributor (including but not limited to Illya Yalovyy, Ying Cao, > Eric > > > Lin, Chris Teoh, Dmitry Zagorulkin), and as a recognition of their > work, > > > I'd like to ensure their commits gets into a release as soon as > possible. > > > Third: recently Anna, Bogi and Szabi were super active on the front of > > > stabilizing and improving the tests, the build system and so, however > > > according to my feelings the current status quo (waiting for a minor > > > version change with bigger changes) holding them back from more > > fundamental > > > improvements (I guess this is the reason why the Gradle/Maven new build > > > sytem is still in progress, and the conversation around feature > branches > > > started). I think it would make a great impact for the whole community > > and > > > the product itself, if we would be able to remove this burden from > their > > > shoulders, and thus they would be able to focus on bigger > > > changes/challenges. > > > > > > Because of the reasons mentioned above, on the top of my existing > duties, > > > I'd like to help the community's life by owning the 1.4.7 release > > process, > > > and starting the release with branching the trunk as of 2017-06-01 > (next > > > Friday) PST 8PM, and finishing the whole process latest by the end of > > June. > > > > > > Of course before starting even creating the related JIRA tasks, plans, > > the > > > branching, etc. I'd like to collect the inputs from all of you if this > > fits > > > your plan/requirements/etc. > > > > > > Looking forward hearing all of your thoughts, concerns, pros and cons. > > > > > > Best regards, > > > Attila Szabo > > > > > > > > > -- > Szabolcs Vasas > Software Engineer > <http://www.cloudera.com> >
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
> On 五月 17, 2017, 11:50 a.m., Attila Szabo wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java > > Lines 90-94 (patched) > > <https://reviews.apache.org/r/58600/diff/1/?file=1696010#file1696010line90> > > > > Please fix indentation + formatting here! Hi Attila Thanks a lot for your detail suggestions Angela > On 五月 17, 2017, 11:50 a.m., Attila Szabo wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java > > Lines 181 (patched) > > <https://reviews.apache.org/r/58600/diff/1/?file=1696010#file1696010line181> > > > > Is this commit call neded here? > > We've just only executed SELECT statement. What kind of changes we'd > > like to commit? Hi Attila Thanks for your suggestion, I have remove this commit and rollback code for "SELECT" statement Angela > On 五月 17, 2017, 11:50 a.m., Attila Szabo wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java > > Lines 372-394 (patched) > > <https://reviews.apache.org/r/58600/diff/1/?file=1696010#file1696010line372> > > > > Could you please check if this has been already solved in other > > connectors? > > > > Getting schema option sounds quite general, might have been implemented > > elsewhere in the codebase. > > > > Would make sense to abstract it out, and just call it from different > > places. > > > > Thanks for checking! Hi Attila Yes, this has been already solved in Postgresql. Which file is suitable to abstrace it out? - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/#review175233 --- On 五月 24, 2017, 9:20 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58600/ > --- > > (Updated 五月 24, 2017, 9:20 a.m.) > > > Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, > Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-1905 > https://issues.apache.org/jira/browse/SQOOP-1905 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1905 : add --schema option for import-all-tables and list-tables > against db2 > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef > src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchema.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58600/diff/2/ > > > Testing > --- > > Mannual UT is passed > > > Thanks, > > Ying Cao > >
Re: Review Request 58600: add --schema option for import-all-tables and list-tables against db2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/ --- (Updated 五月 24, 2017, 9:20 a.m.) Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, Attila Szabo, and Szabolcs Vasas. Changes --- Update patch with Attila's comments Bugs: SQOOP-1905 https://issues.apache.org/jira/browse/SQOOP-1905 Repository: sqoop-trunk Description --- SQOOP-1905 : add --schema option for import-all-tables and list-tables against db2 Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/58600/diff/2/ Changes: https://reviews.apache.org/r/58600/diff/1-2/ Testing --- Mannual UT is passed Thanks, Ying Cao
Re: Review Request 58477: support DB2 xml data type when sqoop import with parquet
> On 五月 4, 2017, 2:18 p.m., Liz Szilagyi wrote: > > Hi Ying Cao, > > Could you please add Sqoop to the reviewer groups? It would make it easier > > to find this review request. > > > > I tried to run > > org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest but it > > failed. > > Could you please clarify how exactly this test should be run or look into > > it to see if some problems accidentally were left in the code? > > > > Thank you, > > Liz Hi Liz, Thanks for your remind, I added Sqoop to "Reviewers Group" now. By the way, this UT is an third-part test, plese try it again by: "ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty -Dsqoop.test.db2.connectstring.host_url=connectstring -Dsqoop.test.db2.connectstring.database=databasename -Dsqoop.test.db2.connectstring.username=username -Dsqoop.test.db2.connectstring.password=password -Dtestcase=DB2XmlTypeParquetImportManualTest" Thanks - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58477/#review173906 --- On 五月 8, 2017, 7:35 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58477/ > --- > > (Updated 五月 8, 2017, 7:35 a.m.) > > > Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-3173 > https://issues.apache.org/jira/browse/SQOOP-3173 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-3173 : support DB2 xml data type when sqoop import with parquet > > > Diffs > - > > src/java/org/apache/sqoop/manager/ConnManager.java 1811ce0 > > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeParquetImportManualTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58477/diff/3/ > > > Testing > --- > > UT passed by manually > > This is third-part test, I ran it by : > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty > -Dsqoop.test.db2.connectstring.host_url=connectstring > -Dsqoop.test.db2.connectstring.database=databasename > -Dsqoop.test.db2.connectstring.username=username > -Dsqoop.test.db2.connectstring.password=password > -Dtestcase=DB2XmlTypeParquetImportManualTest > > > Thanks, > > Ying Cao > >
Re: Review Request 58477: support DB2 xml data type when sqoop import with parquet
bTestCase.runImport(ImportJobTestCase.java:243) > > at > > org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.runDb2Test(DB2XmlTypeParquetImportManualTest.java:241) > > at > > org.apache.sqoop.manager.db2.DB2XmlTypeParquetImportManualTest.testParquetDb2Import(DB2XmlTypeParquetImportManualTest.java:212) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > > at > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > > at > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > > at > > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > > > > Is there any extra setting which should be made before running maybe? I ran > > the command below: > > > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty > > -Dsqoop.test.db2.connectstring.host_url=connectstring > > -Dsqoop.test.db2.connectstring.database=databasename > > -Dsqoop.test.db2.connectstring.username=username > > -Dsqoop.test.db2.connectstring.password=password > > -Dtestcase=DB2XmlTypeParquetImportManualTest > > > > Thanks, > > Bogi Hi Bogi, The error seems happened UT run using invalidate import argvs, but I can not reproduce in my environment. I have update an new patch to LOG the import argvs, it will LOG import args under string of "runImport with argv is" - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58477/#review173726 --- On 五月 8, 2017, 7:35 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58477/ > --- > > (Updated 五月 8, 2017, 7:35 a.m.) > > > Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-3173 > https://issues.apache.org/jira/browse/SQOOP-3173 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-3173 : support DB2 xml data type when sqoop import with parquet > > > Diffs > - > > src/java/org/apache/sqoop/manager/ConnManager.java 1811ce0 > > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeParquetImportManualTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58477/diff/3/ > > > Testing > --- > > UT passed by manually > > This is third-part test, I ran it by : > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty > -Dsqoop.test.db2.connectstring.host_url=connectstring > -Dsqoop.test.db2.connectstring.database=databasename > -Dsqoop.test.db2.connectstring.username=username > -Dsqoop.test.db2.connectstring.password=password > -Dtestcase=DB2XmlTypeParquetImportManualTest > > > Thanks, > > Ying Cao > >
Re: Review Request 58477: support DB2 xml data type when sqoop import with parquet
> On 五月 1, 2017, 4:59 p.m., Boglarka Egyed wrote: > > Hi Ying, > > > > I ran DB2XmlTypeParquetImportManualTest test case added by you and it > > failed with [ERROR -org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:185)] Got > > exception running Sqoop: java.lang.IllegalArgumentException: Cannot convert > > SQL type 2009 > > > > Could you please review your change? > > > > I also left two minor comments on your patch. > > > > Thanks, > > Bogi Hi Bogi, Thanks a lot for your review. I have update patch with fix of your comments, please help to review it again, thanks! By the way I used command "ant clean -Dtestcase=DB2XmlTypeParquetImportManualTest test" to run UT, build successed - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58477/#review173471 ------- On 五月 8, 2017, 7:35 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58477/ > --- > > (Updated 五月 8, 2017, 7:35 a.m.) > > > Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-3173 > https://issues.apache.org/jira/browse/SQOOP-3173 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-3173 : support DB2 xml data type when sqoop import with parquet > > > Diffs > - > > src/java/org/apache/sqoop/manager/ConnManager.java 1811ce0 > > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeParquetImportManualTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58477/diff/3/ > > > Testing > --- > > UT passed by manually > > This is third-part test, I ran it by : > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty > -Dsqoop.test.db2.connectstring.host_url=connectstring > -Dsqoop.test.db2.connectstring.database=databasename > -Dsqoop.test.db2.connectstring.username=username > -Dsqoop.test.db2.connectstring.password=password > -Dtestcase=DB2XmlTypeParquetImportManualTest > > > Thanks, > > Ying Cao > >
Re: Review Request 58477: support DB2 xml data type when sqoop import with parquet
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58477/ --- (Updated 五月 8, 2017, 7:35 a.m.) Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas. Bugs: SQOOP-3173 https://issues.apache.org/jira/browse/SQOOP-3173 Repository: sqoop-trunk Description --- SQOOP-3173 : support DB2 xml data type when sqoop import with parquet Diffs - src/java/org/apache/sqoop/manager/ConnManager.java 1811ce0 src/test/org/apache/sqoop/manager/db2/DB2XmlTypeParquetImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/58477/diff/3/ Testing (updated) --- UT passed by manually This is third-part test, I ran it by : ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdparty -Dsqoop.test.db2.connectstring.host_url=connectstring -Dsqoop.test.db2.connectstring.database=databasename -Dsqoop.test.db2.connectstring.username=username -Dsqoop.test.db2.connectstring.password=password -Dtestcase=DB2XmlTypeParquetImportManualTest Thanks, Ying Cao
[jira] [Commented] (SQOOP-1905) add --schema option for import-all-tables and list-tables against db2
[ https://issues.apache.org/jira/browse/SQOOP-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977979#comment-15977979 ] Ying Cao commented on SQOOP-1905: - [~abec] [~jarcec] [~maugli] I have add test case and create review board entry, please kindly review it, thanks! > add --schema option for import-all-tables and list-tables against db2 > - > > Key: SQOOP-1905 > URL: https://issues.apache.org/jira/browse/SQOOP-1905 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4+Sqoop1.4.5+Hadoop2.4.1+Hive0.13.0+DB2 10.5 >Reporter: xieshiju >Assignee: Ying Cao > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1905.patch, SQOOP-1905.patch, SQOOP-1905.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > For Sqoop import-all-tables, and list-tables function, we can use --schema > option to qualify the table name. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-1905) add --schema option for import-all-tables and list-tables against db2
[ https://issues.apache.org/jira/browse/SQOOP-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1905: Attachment: SQOOP-1905.patch > add --schema option for import-all-tables and list-tables against db2 > - > > Key: SQOOP-1905 > URL: https://issues.apache.org/jira/browse/SQOOP-1905 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4+Sqoop1.4.5+Hadoop2.4.1+Hive0.13.0+DB2 10.5 >Reporter: xieshiju >Assignee: Ying Cao > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1905.patch, SQOOP-1905.patch, SQOOP-1905.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > For Sqoop import-all-tables, and list-tables function, we can use --schema > option to qualify the table name. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 58600: add --schema option for import-all-tables and list-tables against db2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58600/ --- Review request for Sqoop, Abraham Elmahrek, Boglarka Egyed, Jarek Cecho, Attila Szabo, and Szabolcs Vasas. Bugs: SQOOP-1905 https://issues.apache.org/jira/browse/SQOOP-1905 Repository: sqoop-trunk Description --- SQOOP-1905 : add --schema option for import-all-tables and list-tables against db2 Diffs - src/java/org/apache/sqoop/manager/Db2Manager.java 52ab05ef src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/58600/diff/1/ Testing --- Mannual UT is passed Thanks, Ying Cao
[jira] [Commented] (SQOOP-816) Scoop and support for external Hive tables
[ https://issues.apache.org/jira/browse/SQOOP-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972170#comment-15972170 ] Ying Cao commented on SQOOP-816: Hive external table is an very import, it will be very useful for sqoop to support Hive external table > Scoop and support for external Hive tables > -- > > Key: SQOOP-816 > URL: https://issues.apache.org/jira/browse/SQOOP-816 > Project: Sqoop > Issue Type: Improvement > Components: hive-integration >Reporter: Santosh Achhra >Assignee: Chris Teoh >Priority: Minor > Labels: External, Hive,, Scoop,, Tables, newbie > > Sqoop is not supporting HIVE external tables at the moment. Any imports using > scoop creates a managed table, in real world scenario it is very important to > have EXTERNAL tables. As of now we have to execute ALTER statement to change > table properties to make the the table as external table which is not a big > deal but it would nice have an option in scoop to specify type of table which > is required -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SQOOP-3173) support DB2 xml data type when sqoop import with parquet
[ https://issues.apache.org/jira/browse/SQOOP-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972076#comment-15972076 ] Ying Cao commented on SQOOP-3173: - [~maugli] could you kindly review SQOOP-3173.patch I uploaded? > support DB2 xml data type when sqoop import with parquet > > > Key: SQOOP-3173 > URL: https://issues.apache.org/jira/browse/SQOOP-3173 > Project: Sqoop > Issue Type: Bug > Components: connectors >Affects Versions: 1.4.6 > Environment: RedHat6.4+Sqoop 1.4.6 +Hadoop 2.7.3 >Reporter: Ying Cao >Assignee: Ying Cao >Priority: Minor > Labels: patch > Fix For: 1.4.7 > > Attachments: SQOOP-3173.patch > > > When sqoop run import job with parquet from db2, the xml data type failed > with : > 17/01/16 00:48:39 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL > type > Sqoop using Kite SDK to revolve parquet dataset, and it requires an Avro > schema to represent the data structure, during create avro schema sqoop can > not find corresponding type mapping of Types.OTHER. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3173) support DB2 xml data type when sqoop import with parquet
[ https://issues.apache.org/jira/browse/SQOOP-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-3173: Attachment: SQOOP-3173.patch > support DB2 xml data type when sqoop import with parquet > > > Key: SQOOP-3173 > URL: https://issues.apache.org/jira/browse/SQOOP-3173 > Project: Sqoop > Issue Type: Bug > Components: connectors >Affects Versions: 1.4.6 > Environment: RedHat6.4+Sqoop 1.4.6 +Hadoop 2.7.3 >Reporter: Ying Cao >Assignee: Ying Cao >Priority: Minor > Labels: patch > Fix For: 1.4.7 > > Attachments: SQOOP-3173.patch > > > When sqoop run import job with parquet from db2, the xml data type failed > with : > 17/01/16 00:48:39 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL > type > Sqoop using Kite SDK to revolve parquet dataset, and it requires an Avro > schema to represent the data structure, during create avro schema sqoop can > not find corresponding type mapping of Types.OTHER. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3173) support DB2 xml data type when sqoop import with parquet
Ying Cao created SQOOP-3173: --- Summary: support DB2 xml data type when sqoop import with parquet Key: SQOOP-3173 URL: https://issues.apache.org/jira/browse/SQOOP-3173 Project: Sqoop Issue Type: Bug Components: connectors Affects Versions: 1.4.6 Environment: RedHat6.4+Sqoop 1.4.6 +Hadoop 2.7.3 Reporter: Ying Cao Priority: Minor Fix For: 1.4.7 When sqoop run import job with parquet from db2, the xml data type failed with : 17/01/16 00:48:39 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL type Sqoop using Kite SDK to revolve parquet dataset, and it requires an Avro schema to represent the data structure, during create avro schema sqoop can not find corresponding type mapping of Types.OTHER. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Support object store
Hi Expert, Does Sqoop support object store? For example Amazon S3? Angela
[jira] [Assigned] (SQOOP-1905) add --schema option for import-all-tables and list-tables against db2
[ https://issues.apache.org/jira/browse/SQOOP-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao reassigned SQOOP-1905: --- Assignee: Ying Cao (was: xieshiju) > add --schema option for import-all-tables and list-tables against db2 > - > > Key: SQOOP-1905 > URL: https://issues.apache.org/jira/browse/SQOOP-1905 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4+Sqoop1.4.5+Hadoop2.4.1+Hive0.13.0+DB2 10.5 >Reporter: xieshiju >Assignee: Ying Cao > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1905.patch, SQOOP-1905.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > For Sqoop import-all-tables, and list-tables function, we can use --schema > option to qualify the table name. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao reassigned SQOOP-1904: --- Assignee: Ying Cao (was: xieshiju) > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: Ying Cao > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.6.patch, > SQOOP-1904.7.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891491#comment-15891491 ] Ying Cao commented on SQOOP-1904: - [~maugli] could you assign this JIRA to me? By the way I am also interest in other Sqoop JIRAs, could you add me as an contributor? Then I could assign JIRA and work on it. Thanks Angela > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.6.patch, > SQOOP-1904.7.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Help to add me as contributor
Hi all, Anyone could kindly add me as contributor? Thanks Angela
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.7.patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.6.patch, > SQOOP-1904.7.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- (Updated 二月 24, 2017, 6:56 a.m.) Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Changes --- Fix issue base on Szabolcs comments Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860591#comment-15860591 ] Ying Cao commented on SQOOP-1904: - [~maugli] [~abrahamfine] Could you please kindly review SQOOP-1904.6.patch we uploaded? > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.6.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.6.patch Update patch with Abraham Elmahrek comments > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.6.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
> On 二月 8, 2017, 10:25 p.m., Abraham Elmahrek wrote: > > Hi Abraham Elmahrek, Thanks for your comments Angela - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/#review164793 --- On 二月 9, 2017, 3:39 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55132/ > --- > > (Updated 二月 9, 2017, 3:39 a.m.) > > > Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. > > > Bugs: SQOOP-1904 > https://issues.apache.org/jira/browse/SQOOP-1904 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1904: support for DB2 XML data type when importing to hdfs > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/55132/diff/ > > > Testing > --- > > UT passed by manually > > > Thanks, > > Ying Cao > >
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- (Updated 二月 9, 2017, 3:39 a.m.) Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Changes --- Update patch SQOOP-1904.6.patch wtih Abraham's comments Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
> On 二月 8, 2017, 10:25 p.m., Abraham Elmahrek wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java, line 176 > > <https://reviews.apache.org/r/55132/diff/4/?file=1624607#file1624607line176> > > > > Is this a Hive specific type? Or a Java specific type? Hi Abraham Elmahrek, Thanks for your comments. This is Java specific type > On 二月 8, 2017, 10:25 p.m., Abraham Elmahrek wrote: > > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java, > > lines 218-220 > > <https://reviews.apache.org/r/55132/diff/4/?file=1624608#file1624608line218> > > > > Is this necessary? Thanks for your comments. > On 二月 8, 2017, 10:25 p.m., Abraham Elmahrek wrote: > > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java, line > > 215 > > <https://reviews.apache.org/r/55132/diff/4/?file=1624608#file1624608line215> > > > > You could just move the contents of this method into the test case > > method above Thanks for comments! This method will help for resued if other more testing scenario is need. - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/#review164793 --- On 二月 6, 2017, 3:19 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55132/ > --- > > (Updated 二月 6, 2017, 3:19 a.m.) > > > Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. > > > Bugs: SQOOP-1904 > https://issues.apache.org/jira/browse/SQOOP-1904 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1904: support for DB2 XML data type when importing to hdfs > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/55132/diff/ > > > Testing > --- > > UT passed by manually > > > Thanks, > > Ying Cao > >
[jira] [Comment Edited] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853470#comment-15853470 ] Ying Cao edited comment on SQOOP-1904 at 2/6/17 3:40 AM: - Create document JIRA SQOOP-3131 was (Author: ying cao): Create document SQOOP-3131 > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853470#comment-15853470 ] Ying Cao commented on SQOOP-1904: - Create document SQOOP-3131 > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3131) Docuemtn support for DB2 XML data type when importing to hdfs
Ying Cao created SQOOP-3131: --- Summary: Docuemtn support for DB2 XML data type when importing to hdfs Key: SQOOP-3131 URL: https://issues.apache.org/jira/browse/SQOOP-3131 Project: Sqoop Issue Type: Improvement Components: docs Affects Versions: 1.4.5 Environment: RedHat6.4 + Sqoop 1.4.5 + hadoop 2.4.1 Reporter: Ying Cao Priority: Minor Fix For: 1.4.7 After SQOOP-1904, Sqoop should supoort for DB2 XML data type importing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.5.patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.5.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- (Updated 二月 6, 2017, 3:19 a.m.) Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Changes --- Hi Bogi, Thanks for your comments again, name of "JAVA_STRING_DATA_TYPE" is more clean than before. Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
[jira] [Comment Edited] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827536#comment-15827536 ] Ying Cao edited comment on SQOOP-1904 at 1/23/17 6:22 AM: -- [~abec] [~jarcec] [~BoglarkaEgyed] Could you please kindly review SQOOP-1904.3.path we uploaded? was (Author: ying cao): [~abec] [~jarcec] [~BoglarkaEgyed] Could you please kindly review SQOOP-1903.3.path we uploaded? > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794592#comment-15794592 ] Ying Cao edited comment on SQOOP-1904 at 1/23/17 6:23 AM: -- [~jarcec] I create UT for it, and debase a new patch was (Author: ying cao): [~jarcec]I create UT for it, and debase a new patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833956#comment-15833956 ] Ying Cao commented on SQOOP-1904: - [~abec] and [~jarcec] Could you please kindly reivew SQOOP-1904.4.path we uploaded? > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
> On 一月 18, 2017, 5:52 p.m., Boglarka Egyed wrote: > > src/java/org/apache/sqoop/manager/Db2Manager.java, line 53 > > <https://reviews.apache.org/r/55132/diff/2/?file=1604389#file1604389line53> > > > > Thanks for creating a constant for this, however, it would be great to > > check if there is already an enum or something for Hive data types and use > > it if so. Hi Bogi, Sorry to confused, this is an JAVA DATA TYPE, I have update name to JAVA_DATA_TYPE. - Ying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/#review162140 ------- On 一月 23, 2017, 6:17 a.m., Ying Cao wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55132/ > --- > > (Updated 一月 23, 2017, 6:17 a.m.) > > > Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. > > > Bugs: SQOOP-1904 > https://issues.apache.org/jira/browse/SQOOP-1904 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-1904: support for DB2 XML data type when importing to hdfs > > > Diffs > - > > src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c > src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/55132/diff/ > > > Testing > --- > > UT passed by manually > > > Thanks, > > Ying Cao > >
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.4.patch generate SQOOP-1904.4.patch base on latest trunk > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, > SQOOP-1904.4.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- (Updated 一月 23, 2017, 6:17 a.m.) Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Changes --- Hi Bogi, Thanks for your comments. I have rebased my patch with latest trunk and with fix base on your comments, please kindly review it again, thanks! Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827536#comment-15827536 ] Ying Cao commented on SQOOP-1904: - [~abec] [~jarcec] [~BoglarkaEgyed] Could you please kindly review SQOOP-1903.3.path we uploaded? > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.3.patch Update patch base on Bogi comments > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-2894) Hive import with Parquet failed in Kerberos enabled cluster
[ https://issues.apache.org/jira/browse/SQOOP-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823391#comment-15823391 ] Ying Cao commented on SQOOP-2894: - Hi ping, This issue is fixed by SQOOP-2406:Add support for secure mode when importing Parquet files into Hive > Hive import with Parquet failed in Kerberos enabled cluster > --- > > Key: SQOOP-2894 > URL: https://issues.apache.org/jira/browse/SQOOP-2894 > Project: Sqoop > Issue Type: Bug > Components: hive-integration, tools >Affects Versions: 1.4.6 > Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 >Reporter: Ping Wang > Labels: security > Fix For: 1.4.7 > > > Importing data from external database to hive with Parquet option failed in > the kerberos environment. (It can success without kerberos). > The sqoop command I used: > sqoop import --connect jdbc:db2://xxx:5/testdb --username xxx --password > xxx --table users --hive-import -hive-table users3 --as-parquetfile -m 1 > The import job failed: > .. > 2016-02-26 04:20:07,020 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. > 2016-02-26 04:20:08,088 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config > null > 2016-02-26 04:20:08,918 INFO [main] hive.metastore: Trying to connect to > metastore with URI thrift://xxx:9083 > 2016-02-26 04:30:09,207 WARN [main] hive.metastore: set_ugi() not successful, > Likely cause: new client talking to old server. Continuing without it. > org.apache.thrift.transport.TTransportException: > java.net.SocketTimeoutException: Read timed out > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:448) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:237) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:182) > at org.kitesdk.data.spi.hive.MetaStoreUtil.(MetaStoreUtil.java:82) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.getMetaStoreUtil(HiveAbstractMetadataProvider.java:63) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:270) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:255) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:102) > at > org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192) > at org.kitesdk.data.Datasets.load(Datasets.java:108) > at org.kitesdk.data.Datasets.load(Datasets.java:165) > at > org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510) > at > org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- (Updated 一月 13, 2017, 9:46 a.m.) Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Changes --- Hi Bogi, Thanks for you comments, I have fix them and update diff file, please help review it again Thanks Angela Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs (updated) - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/com/cloudera/sqoop/ThirdPartyTests.java 7e10c68 src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
Review Request 55132: support for DB2 XML data type when importing to hdfs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55132/ --- Review request for Sqoop, Abraham Elmahrek and Jarek Cecho. Bugs: SQOOP-1904 https://issues.apache.org/jira/browse/SQOOP-1904 Repository: sqoop-trunk Description --- SQOOP-1904: support for DB2 XML data type when importing to hdfs Diffs - src/java/org/apache/sqoop/manager/Db2Manager.java e39aa4c src/test/com/cloudera/sqoop/ThirdPartyTests.java 7e10c68 src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55132/diff/ Testing --- UT passed by manually Thanks, Ying Cao
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.2.patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: (was: 0001-SQOOP-1904.patch) > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794592#comment-15794592 ] Ying Cao commented on SQOOP-1904: - [~jarcec]I create UT for it, and debase a new patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: 0001-SQOOP-1904.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: 0001-SQOOP-1904.patch > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: 0001-SQOOP-1904.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3059) sqoop import into hive converting decimals to decimals
[ https://issues.apache.org/jira/browse/SQOOP-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707481#comment-15707481 ] Ying Cao commented on SQOOP-3059: - [~bleeapache] [~usblee] I found the decimal type mapping is invoked by SQOOP-380, do you have any suggestion for mapping of decimal? By default it will converting to double. Thanks > sqoop import into hive converting decimals to decimals > -- > > Key: SQOOP-3059 > URL: https://issues.apache.org/jira/browse/SQOOP-3059 > Project: Sqoop > Issue Type: Improvement > Components: hive-integration > Environment: Redhat 6.6, Sqoop 1.4.6 >Reporter: Ying Cao >Priority: Minor > > By default sqoop import into hive converting decimals to double, Decimal type > provide more accurate values wider range than Double, so by default way some > data will be truncated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-3059) sqoop import into hive converting decimals to decimals
Ying Cao created SQOOP-3059: --- Summary: sqoop import into hive converting decimals to decimals Key: SQOOP-3059 URL: https://issues.apache.org/jira/browse/SQOOP-3059 Project: Sqoop Issue Type: Improvement Components: hive-integration Environment: Redhat 6.6, Sqoop 1.4.6 Reporter: Ying Cao Priority: Minor By default sqoop import into hive converting decimals to double, Decimal type provide more accurate values wider range than Double, so by default way some data will be truncated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-3054) Get FileSystem from parameter "--target-dir"
[ https://issues.apache.org/jira/browse/SQOOP-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-3054: Priority: Critical (was: Major) > Get FileSystem from parameter "--target-dir" > > > Key: SQOOP-3054 > URL: https://issues.apache.org/jira/browse/SQOOP-3054 > Project: Sqoop > Issue Type: Bug > Components: connectors >Affects Versions: 1.4.6 > Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 >Reporter: Ying Cao >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3054.patch > > > FileSystems gets from conf , update to get filesystem from parateter > "--target-dir" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3054) Get FileSystem from parameter "--target-dir"
[ https://issues.apache.org/jira/browse/SQOOP-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685894#comment-15685894 ] Ying Cao commented on SQOOP-3054: - getFileSystem() from "--target-dir" > Get FileSystem from parameter "--target-dir" > > > Key: SQOOP-3054 > URL: https://issues.apache.org/jira/browse/SQOOP-3054 > Project: Sqoop > Issue Type: Bug > Components: connectors >Affects Versions: 1.4.6 > Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 >Reporter: Ying Cao > Fix For: 1.4.7 > > Attachments: SQOOP-3054.patch > > > FileSystems gets from conf , update to get filesystem from parateter > "--target-dir" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-3054) Get FileSystem from parameter "--target-dir"
[ https://issues.apache.org/jira/browse/SQOOP-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-3054: Attachment: SQOOP-3054.patch > Get FileSystem from parameter "--target-dir" > > > Key: SQOOP-3054 > URL: https://issues.apache.org/jira/browse/SQOOP-3054 > Project: Sqoop > Issue Type: Bug > Components: connectors >Affects Versions: 1.4.6 > Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 >Reporter: Ying Cao > Fix For: 1.4.7 > > Attachments: SQOOP-3054.patch > > > FileSystems gets from conf , update to get filesystem from parateter > "--target-dir" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-3054) Get FileSystem from parameter "--target-dir"
Ying Cao created SQOOP-3054: --- Summary: Get FileSystem from parameter "--target-dir" Key: SQOOP-3054 URL: https://issues.apache.org/jira/browse/SQOOP-3054 Project: Sqoop Issue Type: Bug Components: connectors Affects Versions: 1.4.6 Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 Reporter: Ying Cao Fix For: 1.4.7 FileSystems gets from conf , update to get filesystem from parateter "--target-dir" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
timeline for Sqoop 1.4.7
Hello everyone, Latest stable version of Sqoop is 1.4.6, is there any timeline for next version 1.4.7? I have check Sqoop 1.4.7 JIRA with following URI, https://issues.apache.org/jira/browse/SQOOP/fixforversion/12329683/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel Sqoop 1.4.7 is between 1.99.7 and 1.99.8, does it means 1.4.7 is next release version after 1.99.7, and 1.99.8 release is after 1.4.7?
Sqoop branch version
Hellow everyone, I find there five kinds of sqoop branch: 1.trunk 2.branch 1.4.* 3.branch 1.99.* 4. sqoop 5. sqoop2 I want to check Sqoop 1.4.7, but I can not find "branch 1.4.7", with my understanding as sqoop 1.4.7 does not release so it have not "branch 1.4.7". But where could I find code of 1.4.7. More all 1. #4 is handle sqoop 1 code , 2. #5 handle sqoop 2 code, 3. what's different between #2 ,#3 and #4 4. how about trunk branch. Thanks & Regards Angela
