[jira] [Commented] (SQOOP-2903) Add Kudu connector for Sqoop

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908139#comment-16908139
 ] 

Szabolcs Vasas commented on SQOOP-2903:
---

Hi [~tsss686],

This feature was never committed so it is not available in Sqoop yet.
Since the patch was updated years ago and it conflicts with the latest codebase 
I suspect it requires a significant amount of work to make it work.

Regards,
Szabolcs

> Add Kudu connector for Sqoop
> 
>
> Key: SQOOP-2903
> URL: https://issues.apache.org/jira/browse/SQOOP-2903
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors
>Reporter: Sameer Abhyankar
>Assignee: Sameer Abhyankar
>Priority: Major
> Attachments: SQOOP-2903.1.patch, SQOOP-2903.2.patch, SQOOP-2903.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Sqoop currently does not have a connector for Kudu. We should add the 
> functionality to allow Sqoop to ingest data directly into Kudu.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (SQOOP-3134) --class-name should override default Avro schema name

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3134.
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   1.5.0

Hi [~dvoros],

Your patch is now committed, thank you for your contribution!

Regards,
Szabolcs

> --class-name should override default Avro schema name
> -
>
> Key: SQOOP-3134
> URL: https://issues.apache.org/jira/browse/SQOOP-3134
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Markus Kemper
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3134.1.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Please consider adding an option to configure the Avro schema output file 
> name that is created with Sqoop (import + --as-avrodatafile), example cases 
> below.
> {noformat}
> #
> # STEP 01 - Create Data
> #
> export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
> export MYUSER=sqoop
> export MYPSWD=cloudera
> sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "create table t1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1"
> -
> | c1  | c2 | c3 | 
> -
> | 1   | 2017-02-13 | some data  | 
> -
> #
> # STEP 02 - Import + --table + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 
> --as-avrodatafile 
> ls -l ./*
> Output:
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in 
> 20.6988 seconds (19.9529 bytes/sec)
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> 
> -rw-r--r-- 1 root root   492 Feb 13 12:14 ./t1.avsc < want option to 
> configure this file name
> -rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java
> #
> # STEP 03 - Import + --query + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir 
> /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
> ls -l ./*
> Output:
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in 
> 25.2757 seconds (17.7245 bytes/sec)
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> -rw-r--r-- 1 root root   527 Feb 13 12:16 ./AutoGeneratedSchema.avsc < 
> want option to configure this file name
> -rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3134) --class-name should override default Avro schema name

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3134:
--
Summary: --class-name should override default Avro schema name  (was: Add 
option to configure Avro schema output file name with (import + 
--as-avrodatafile) )

> --class-name should override default Avro schema name
> -
>
> Key: SQOOP-3134
> URL: https://issues.apache.org/jira/browse/SQOOP-3134
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Markus Kemper
>Assignee: Daniel Voros
>Priority: Major
> Attachments: SQOOP-3134.1.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Please consider adding an option to configure the Avro schema output file 
> name that is created with Sqoop (import + --as-avrodatafile), example cases 
> below.
> {noformat}
> #
> # STEP 01 - Create Data
> #
> export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
> export MYUSER=sqoop
> export MYPSWD=cloudera
> sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "create table t1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1"
> -
> | c1  | c2 | c3 | 
> -
> | 1   | 2017-02-13 | some data  | 
> -
> #
> # STEP 02 - Import + --table + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1 
> --as-avrodatafile 
> ls -l ./*
> Output:
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in 
> 20.6988 seconds (19.9529 bytes/sec)
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> 
> -rw-r--r-- 1 root root   492 Feb 13 12:14 ./t1.avsc < want option to 
> configure this file name
> -rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java
> #
> # STEP 03 - Import + --query + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir 
> /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile 
> ls -l ./*
> Output:
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in 
> 25.2757 seconds (17.7245 bytes/sec)
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> -rw-r--r-- 1 root root   527 Feb 13 12:16 ./AutoGeneratedSchema.avsc < 
> want option to configure this file name
> -rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3435) Avoid NullPointerException due to different JSONObject library in classpath

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3435:
-

Assignee: Tak Lon (Stephen) Wu

> Avoid NullPointerException due to different JSONObject library in classpath
> ---
>
> Key: SQOOP-3435
> URL: https://issues.apache.org/jira/browse/SQOOP-3435
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: SQOOP-3435.trunk.001.patch, SQOOP-3435.trunk.002.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In line of 
> [SqoopOptions.java#L785|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L785]
>  with
> {\{SqoopJsonUtil.getJsonStringforMap((Map) f.get(this))); }}
> Above line should check NULL pointer like in line of 
> [SqoopOptions.java#L778|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L778]
>  which has
> {{f.get(this) == null ? "null" : f.get(this).toString()}}
> Please see the stacktrace below when running command:
> {{sqoop job --create myjob -- import --connect jdbc:mysql://localhost/db 
> --username root --table employee --m 1}}
> {code:java}
> 19/02/02 01:09:21 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.json.JSONObject.(JSONObject.java:144)
> at 
> org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
> at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
> at 
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
> at 
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
> at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> The above {{NullPointerException}} is due to use the of 
> [{{com.tdunning}}|https://github.com/tdunning/open-json/blob/rc1.8/src/main/java/org/json/JSONObject.java#L141-L155]
>  as part of the HIVE libs (if one is reusing the {{HADOOP_CLASSPATH}}) in the 
> classpath. but I think we can better have a checker of {{null}} in 
> {{SqoopJsonUtil.getJsonStringforMap(Map map)}} before calling 
> {{JSONObject pathPartMap = new JSONObject(map);}}
> Reporting this bug and the right behavior need to be decided by the assignee. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3435) Avoid NullPointerException due to different JSONObject library in classpath

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3435:
--
Summary: Avoid NullPointerException due to different JSONObject library in 
classpath  (was: SqoopOptions missed a null check)

> Avoid NullPointerException due to different JSONObject library in classpath
> ---
>
> Key: SQOOP-3435
> URL: https://issues.apache.org/jira/browse/SQOOP-3435
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: SQOOP-3435.trunk.001.patch, SQOOP-3435.trunk.002.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In line of 
> [SqoopOptions.java#L785|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L785]
>  with
> {\{SqoopJsonUtil.getJsonStringforMap((Map) f.get(this))); }}
> Above line should check NULL pointer like in line of 
> [SqoopOptions.java#L778|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L778]
>  which has
> {{f.get(this) == null ? "null" : f.get(this).toString()}}
> Please see the stacktrace below when running command:
> {{sqoop job --create myjob -- import --connect jdbc:mysql://localhost/db 
> --username root --table employee --m 1}}
> {code:java}
> 19/02/02 01:09:21 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.json.JSONObject.(JSONObject.java:144)
> at 
> org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
> at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
> at 
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
> at 
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
> at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> The above {{NullPointerException}} is due to use the of 
> [{{com.tdunning}}|https://github.com/tdunning/open-json/blob/rc1.8/src/main/java/org/json/JSONObject.java#L141-L155]
>  as part of the HIVE libs (if one is reusing the {{HADOOP_CLASSPATH}}) in the 
> classpath. but I think we can better have a checker of {{null}} in 
> {{SqoopJsonUtil.getJsonStringforMap(Map map)}} before calling 
> {{JSONObject pathPartMap = new JSONObject(map);}}
> Reporting this bug and the right behavior need to be decided by the assignee. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3427) Add Travis badge to the Readme

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3427.
---
Resolution: Fixed

HI [~Fokko],

Your patch is now committed, thank you for your contribution!



> Add Travis badge to the Readme
> --
>
> Key: SQOOP-3427
> URL: https://issues.apache.org/jira/browse/SQOOP-3427
> Project: Sqoop
>  Issue Type: Improvement
>  Components: docs
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We love badges



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3429) Bump Hadoop to 2.9.2

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3429.
---
Resolution: Fixed

Hi [~Fokko],

Your patch is now committed, thank you for your contribution!

> Bump Hadoop to 2.9.2
> 
>
> Key: SQOOP-3429
> URL: https://issues.apache.org/jira/browse/SQOOP-3429
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I would like to bump Sqoop to Hadoop 2.9.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3429) Bump Hadoop to 2.9.2

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3429:
--
Summary: Bump Hadoop to 2.9.2  (was: Bump Sqoop to Hadoop 2.9.2)

> Bump Hadoop to 2.9.2
> 
>
> Key: SQOOP-3429
> URL: https://issues.apache.org/jira/browse/SQOOP-3429
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I would like to bump Sqoop to Hadoop 2.9.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3429) Bump Sqoop to Hadoop 2.9.x

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3429:
-

Assignee: Fokko Driesprong

> Bump Sqoop to Hadoop 2.9.x
> --
>
> Key: SQOOP-3429
> URL: https://issues.apache.org/jira/browse/SQOOP-3429
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I would like to bump Sqoop to Hadoop 2.9.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3429) Bump Sqoop to Hadoop 2.9.2

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3429:
--
Summary: Bump Sqoop to Hadoop 2.9.2  (was: Bump Sqoop to Hadoop 2.9.x)

> Bump Sqoop to Hadoop 2.9.2
> --
>
> Key: SQOOP-3429
> URL: https://issues.apache.org/jira/browse/SQOOP-3429
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I would like to bump Sqoop to Hadoop 2.9.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3430) Remove the old maven pom

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3430.
---
   Resolution: Fixed
Fix Version/s: 1.5.0

Hi [~Fokko],

Your patch is now committed, thank you for your contribution!

> Remove the old maven pom
> 
>
> Key: SQOOP-3430
> URL: https://issues.apache.org/jira/browse/SQOOP-3430
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I think both ant and gradle are enough :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3420) Invalid ERROR message initiates false alarms

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3420:
-

Assignee: Denes Bodo

> Invalid ERROR message initiates false alarms
> 
>
> Key: SQOOP-3420
> URL: https://issues.apache.org/jira/browse/SQOOP-3420
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: SQOOP-3420_001.patch
>
>
> In SQOOP-3042, a debug message was refactored to be error instead means false 
> alarms in customer log analyser. After understanding the functionality it is 
> recommended to use info level message instead of error in case when 
> ImportTool is unable to backup generated .java file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3423) Let user pass password to connect Hive when it set to LDAP authentication

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758291#comment-16758291
 ] 

Szabolcs Vasas commented on SQOOP-3423:
---

Hi [~dionusos],

The direction seems to be correct, feel free to create a PR for it.



> Let user pass password to connect Hive when it set to LDAP authentication
> -
>
> Key: SQOOP-3423
> URL: https://issues.apache.org/jira/browse/SQOOP-3423
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hive-integration
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Major
> Attachments: SQOOP-3423-001.patch
>
>
> If HiveServer2 is using password based authentication, additional 
> username/password information has to be provided to be able to connect to it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3420) Invalid ERROR message initiates false alarms

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758118#comment-16758118
 ] 

Szabolcs Vasas commented on SQOOP-3420:
---

[~dionusos]

Yes, don't worry about the Oracle test failures that happens because of a test 
infra problem we need to address separately.
I have approved the request, if no other reviews will come in I will merge it 
by the end of today.

> Invalid ERROR message initiates false alarms
> 
>
> Key: SQOOP-3420
> URL: https://issues.apache.org/jira/browse/SQOOP-3420
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: SQOOP-3420_001.patch
>
>
> In SQOOP-3042, a debug message was refactored to be error instead means false 
> alarms in customer log analyser. After understanding the functionality it is 
> recommended to use info level message instead of error in case when 
> ImportTool is unable to backup generated .java file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3422) Close database connection in Sqoop CLI before starting import/export job

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3422:
-

 Summary: Close database connection in Sqoop CLI before starting 
import/export job
 Key: SQOOP-3422
 URL: https://issues.apache.org/jira/browse/SQOOP-3422
 Project: Sqoop
  Issue Type: Improvement
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


Sqoop does not close the open database connection before submitting the MR job 
but in most of the cases (except Hive import) it does not need it after the job 
finishes.
The open connection wastes resources and can cause misleading error messages in 
the log if it times out.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3420) Invalid ERROR message initiates false alarms

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757102#comment-16757102
 ] 

Szabolcs Vasas commented on SQOOP-3420:
---

Hi [~dionusos],

It looks good, I am also OK with the WARN log level, please create a pull 
request with the change and we will ask [~ericlin] to review it as well.


> Invalid ERROR message initiates false alarms
> 
>
> Key: SQOOP-3420
> URL: https://issues.apache.org/jira/browse/SQOOP-3420
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: SQOOP-3420_001.patch
>
>
> In SQOOP-3042, a debug message was refactored to be error instead means false 
> alarms in customer log analyser. After understanding the functionality it is 
> recommended to use info level message instead of error in case when 
> ImportTool is unable to backup generated .java file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3421) Importing data from Oracle to Parquet as incremental dataset name fails

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747986#comment-16747986
 ] 

Szabolcs Vasas commented on SQOOP-3421:
---

Hi [~dmateusp],

You have encountered a Kite limitation here. The problem is that since the 
table name is specified in SOME_SCHEMA.SOME_TABLE_NAME form Kite tries to 
create a dataset with that name but '.' is not permitted in Kite dataset names. 
The reason you get this error with Parquet file format only is that Kite was 
only used for Parquet reading/writing.
Kite dependency has been removed from Sqoop a couple of months ago so this 
issue is resolved in the latest trunk but unfortunately we do not have any 
releases yet which contain the fix.

Btw s3n file system is not deprecated you might want to use s3a in the future.

Regards,
Szabolcs

> Importing data from Oracle to Parquet as incremental dataset name fails
> ---
>
> Key: SQOOP-3421
> URL: https://issues.apache.org/jira/browse/SQOOP-3421
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Mateus Pires
>Priority: Minor
>
> Hi there, I'm trying to run the following to import an Oracle table into S3 
> as Parquet:
> {code:bash}
> sqoop import --connect jdbc:oracle:thin:@//some.host:1521/ORCL 
> --where="rownum < 100" --table SOME_SCHEMA.SOME_TABLE_NAME --password 
> some_password --username some_username --num-mappers 4 --split-by PRD_ID 
> --target-dir s3n://bucket/destination --temporary-rootdir 
> s3n://bucket/temp/destination --compress --check-column PRD_MODIFY_DT 
> --incremental lastmodified --map-column-java PRD_ATTR_TEXT=String --append
> {code}
> Version of Kite is: kite-data-s3-1.1.0.jar
> Version of Sqoop is: 1.4.7
> And I'm getting the following error:
> {code:text}
> 19/01/21 13:20:33 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM SOME_SCHEMA.SOME_TABLE_NAME t WHERE 1=0
> 19/01/21 13:20:34 INFO conf.HiveConf: Found configuration file 
> file:/etc/hive/conf.dist/hive-site.xml
> 19/01/21 13:20:35 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> org.kitesdk.data.ValidationException: Dataset name 
> 47a2cf963b82475d8eba78c822403204_SOME_SCHEMA.SOME_TABLE_NAME is not 
> alphanumeric (plus '_')
> org.kitesdk.data.ValidationException: Dataset name 
> 47a2cf963b82475d8eba78c822403204_SOME_SCHEMA.SOME_TABLE_NAME is not 
> alphanumeric (plus '_')
>   at 
> org.kitesdk.data.ValidationException.check(ValidationException.java:55)
>   at 
> org.kitesdk.data.spi.Compatibility.checkDatasetName(Compatibility.java:105)
>   at org.kitesdk.data.spi.Compatibility.check(Compatibility.java:68)
>   at 
> org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.create(FileSystemMetadataProvider.java:209)
>   at 
> org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.create(FileSystemDatasetRepository.java:137)
>   at org.kitesdk.data.Datasets.create(Datasets.java:239)
>   at org.kitesdk.data.Datasets.create(Datasets.java:307)
>   at 
> org.apache.sqoop.mapreduce.ParquetJob.createDataset(ParquetJob.java:156)
>   at 
> org.apache.sqoop.mapreduce.ParquetJob.configureImportJob(ParquetJob.java:130)
>   at 
> org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:132)
>   at 
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:264)
>   at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
>   at 
> org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:454)
>   at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
>   at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> Importing as text file instead solves the issue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746220#comment-16746220
 ] 

Szabolcs Vasas commented on SQOOP-3042:
---

Hi [~vasubramanian],

This makes complete sense, I am happy to help committing a patch to restore the 
original logging behavior, [~dionusos] do you want to prepare a patch for this?

[~ericlin], feel free to chime in if you have anything to add.

Thanks,

Szabolcs

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3419) Refactor S3 tests for easier extensibility

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3419:
-

Assignee: Szabolcs Vasas

> Refactor S3 tests for easier extensibility
> --
>
> Key: SQOOP-3419
> URL: https://issues.apache.org/jira/browse/SQOOP-3419
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> It would be great to add Azure cloud storage tests similar to the S3 tests we 
> have currently but Azure support is implemented in Hadoop 3.x modules only.
> However as a first step we could refactor our existing tests to enable better 
> extensibility in the future.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3419) Refactor S3 tests for easier extensibility

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3419:
-

 Summary: Refactor S3 tests for easier extensibility
 Key: SQOOP-3419
 URL: https://issues.apache.org/jira/browse/SQOOP-3419
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


It would be great to add Azure cloud storage tests similar to the S3 tests we 
have currently but Azure support is implemented in Hadoop 3.x modules only.

However as a first step we could refactor our existing tests to enable better 
extensibility in the future.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740394#comment-16740394
 ] 

Szabolcs Vasas commented on SQOOP-3042:
---

The error means that the generated source file cannot be moved to the code 
output directory, which is the current directory by default but can be 
overridden with the --outdir option.

So depending on the use case it could be a real error because I can imagine in 
some cases the user needs these generated files but in most of the cases they 
care about the import only which is not impacted by this error.

So we could argue for both keeping this behavior and restore the original one 
but I think if you explain the impact of the error to the customer they should 
be ok.

 

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739556#comment-16739556
 ] 

Szabolcs Vasas commented on SQOOP-3042:
---

Hi [~dionusos],

I am not sure it was intentional, does it cause any problem on your side? If 
yes, I think we can submit a patch to restore the original behavior.

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 62523: SQOOP-3237: Mainframe FTP transfer option to insert custom FTP commands prior to transfer

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62523/#review211139
---


Ship it!




Ship It!

- Szabolcs Vasas


On Dec. 10, 2018, 5:45 a.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62523/
> ---
> 
> (Updated Dec. 10, 2018, 5:45 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3237
> https://issues.apache.org/jira/browse/SQOOP-3237
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Added --ftpcmds command to allow comma separated list of FTP commands to send.
> 
> 
> Diffs
> -
> 
>   src/docs/user/import-mainframe.txt 3ecfb7e4 
>   src/java/org/apache/sqoop/SqoopOptions.java f06872f9 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java 
> 9842daa6 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 
> 90dc2ddd 
>   src/java/org/apache/sqoop/tool/MainframeImportTool.java fbc8c3db 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java e7c48a6b 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
>  502e6333 
>   src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 00e57bd0 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java fc6e56d6 
> 
> 
> Diff: https://reviews.apache.org/r/62523/diff/11/
> 
> 
> Testing
> ---
> 
> Unit tests.
> 
> 
> File Attachments
> 
> 
> SQOOP-3237-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/09/26/56041556-e355-4372-83ab-1bcc01680201__SQOOP-3237-1.patch
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas]

Hi Attila,

Thanks for raising this, I think it is a great idea to migrate as soon as
possible, I have started a separate mail chain which can be used as a proof
of community consensus.

On Sun, Dec 9, 2018 at 9:48 PM Attila Szabó  wrote:

> Hello everyone,
>
> So if I'm not mistaken and according to the INFRA mail we've received
> during the weekend gitbox.apache.org is exactly what we wanted to have :
> Having both ASF commits and push privileges on Github side..
> We would just need to have an official consensus in the dev community (
> documented on the mail list), fire an INFRA Jira ticket, and after the
> merge update our site with the new "how to contribute" information.
>
> I think this would be a quite good timing for the Sqoop project as we're
> right now in the middle of these infrastructural reworks.
>
> Kind regards,
> Attila
>
> On Fri, Nov 30, 2018, 2:38 PM Szabolcs Vasas 
> > Thanks for the question, I think we should stick to the commit format we
> > had earlier so yes, we should go for squash and push.
> > The easiest solution could be to download the diff file instead of the
> > patch (e.g. https://github.com/apache/sqoop/pull/60.diff instead of
> > https://github.com/apache/sqoop/pull/60.patch) and that does the trick.
> >
> > The "This closes..." commit message just closes the PR but does not
> delete
> > the feature branch, asfgit most probably does not have delete permission
> > for these branches anyway.
> >
> >
> > On Fri, Nov 30, 2018 at 11:45 AM Attila Szabó  wrote:
> >
> > > Hey Szabi,
> > >
> > > Thanks for the prompt response!
> > >
> > > I've thought the repos are connected back and forth and the close PR is
> > the
> > > official way. In case of we still stack to the patch file and git apply
> > > then commit and push approach.
> > >
> > > My only question in this case :
> > > Do we have any agreement on how we commit these PRs. I would vote for
> > > Squash and push, but of course I'm open for the discussion.
> > >
> > > BTW :
> > > Is "This closes" message deletes the branch automatically?
> > >
> > > On front of Github + Jira:
> > > I'm aware Github has the feature to connect with Jira so I'm pretty
> sure
> > it
> > > doable. Also I'm not sure if any ASF project has done it ever, but I'll
> > ask
> > > around in other communities.
> > >
> > > Cheers,
> > > Attila
> > >
> > >
> > >
> > > On Nov 30, 2018 11:03 AM, "Szabolcs Vasas"  wrote:
> > >
> > > Hi Attila,
> > >
> > > I think we won't be able to commit the pull requests on GitHub directly
> > > because our GitHub repo is just a mirror of the original Apache Sqoop
> > repo
> > > <https://git-wip-us.apache.org/repos/asf/sqoop.git>.
> > > However the commit process is still simplified, the ASF GitHub Bot JIRA
> > > comment contains the patch download link (e.g.
> > > https://github.com/apache/sqoop/pull/60.patch) and the commit message
> > > (e.g. This
> > > closes #60) you need to include to close the pull request. The rest of
> > the
> > > process remains the same, you need to apply the patch manually and push
> > the
> > > commit to git-wip-us repo.
> > >
> > > Regarding the GitHub UI improvement: I am not sure GitHub provides
> such a
> > > feature, do you know an example repository where this is implemented?
> > >
> > > Regards,
> > > Szabolcs
> > >
> > >
> > >
> > > On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó 
> wrote:
> > >
> > > > Hi Szabi,
> > > >
> > > > First of all:
> > > > Big Kudos for the more mature gradle build! I think this is a great
> > step
> > > > for the whole project!
> > > >
> > > > On the front of PRs:
> > > > I would only make it official if the user management / authorization
> > > > handling could be somehow automatically bound to the id.apache.org +
> > > > project privileges.
> > > > A good example:
> > > > Today I reviewed SQOOP-3396 but I would not had been able to merge it
> > > > because it seems on the Github project I do not have push / merge
> > > > privileges (regardless that I'm a Sqoop committer and also memeber of
> > the
> > > > ASF group on github with my user).
> > > > So if we can somehow bound these things together

Mandatory relocation of Sqoop git repository to gitbox

[Szabolcs Vasas]

Hi All,

According to this

email
the git-wip-us server, which hosts Apache Sqoop git repository too, is
going to be decommissioned soon and all the projects are going to be
migrated to gitbox.
For the detailed description of the planned change please refer to the
email linked, but the bottom line is that after the migration we are going
to be able to merge pull requests on the GitHub UI as well which will
greatly simplify our commit process.

This relocation is mandatory however we have the option execute it in the
initial phase which would be great in my opinion because we could start
enjoying the benefits very soon.

Please reply to this chain with your opinion because we need a consensus to
be able to start the migration in the initial phase.

Thanks and regards,
Szabolcs

[jira] [Resolved] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3396.
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   1.5.0

Hi [~fero],

Your patch is now committed, thank you for your contribution!

 

> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3149) Sqoop incremental import - NULL column updates are not pulled into HBase table

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709729#comment-16709729
 ] 

Szabolcs Vasas commented on SQOOP-3149:
---

Hi [~anjaiahspr],

I think you have encountered the issue reported in this JIRA: 
https://issues.apache.org/jira/browse/SQOOP-3267

It is not yet released upstream but might have included in an HDP release, you 
could check with your account manager.

Regards,

Szabolcs

> Sqoop incremental import -  NULL column updates are not pulled into HBase 
> table
> ---
>
> Key: SQOOP-3149
> URL: https://issues.apache.org/jira/browse/SQOOP-3149
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/generic, hbase-integration
>Affects Versions: 1.4.6
>Reporter: Jilani Shaik
>Priority: Major
> Fix For: 1.4.7
>
> Attachments: hbase_delete_support_in_incremental_import
>
>
> Sqoop incremental import data from any database to HBase, if source table's 
> column from a row is updated to NULL, then target HBase table still showing 
> the previous value for that column. 
> So if you do a scan on the table for that row, HBase shows the previous 
> values of the column.
> Expected Result: Sqoop incremental import, If NULL columns are there in 
> source, then HBase need not store that and if it already exists need to 
> delete that column for a given row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3417) Execute Oracle XE tests on Travis CI

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3417:
-

Assignee: Szabolcs Vasas

> Execute Oracle XE tests on Travis CI
> 
>
> Key: SQOOP-3417
> URL: https://issues.apache.org/jira/browse/SQOOP-3417
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> The task is to enable the Travis CI to execute Oracle XE tests too 
> automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3417) Execute Oracle XE tests on Travis CI

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3417:
-

 Summary: Execute Oracle XE tests on Travis CI
 Key: SQOOP-3417
 URL: https://issues.apache.org/jira/browse/SQOOP-3417
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


The task is to enable the Travis CI to execute Oracle XE tests too 
automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3393) TestNetezzaExternalTableExportMapper hangs

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3393.
---
Resolution: Fixed

Hi Daniel,

Your patch is now committed, thank you for your contribution!

> TestNetezzaExternalTableExportMapper hangs
> --
>
> Key: SQOOP-3393
> URL: https://issues.apache.org/jira/browse/SQOOP-3393
> Project: Sqoop
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.5.0, 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
>
> Introduced in SQOOP-3378, spotted by [~vasas].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3415) Fix gradle test+build when clean applied as the first command + warning issue fixes

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3415.
---
Resolution: Fixed

Hi Attila,

Your patch is now committed, thank you for your contribution!

 

> Fix gradle test+build when clean applied as the first command + warning issue 
> fixes
> ---
>
> Key: SQOOP-3415
> URL: https://issues.apache.org/jira/browse/SQOOP-3415
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Attila Szabo
>Assignee: Attila Szabo
>Priority: Major
> Fix For: 1.5.0
>
>
> If the user wants to build like the following command:
> gradlew clean unittest
> the gradle process ends up in an exception and the whole process left there 
> hanging forever. The root cause of this is the following:
> tasks.withType runs in the configuration part of the build, where we ensure 
> the neccessary directories exist.
> after that clean is executed and all of the dirs got deleted.
> Proposed fix:
> Apply directory creation as the first step of test tasks.
> on the top:
> there are some missing options b/c of Junit annotation processors, and also 
> Xlint information are swallowed currently. We aim to fix these things as well



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 62523: SQOOP-3237: Mainframe FTP transfer option to insert custom FTP commands prior to transfer

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62523/#review210979
---



Hi Chris,

Thank you for the improvements!
I think the patch will be OK, I have found some minor issues, please see them 
below.


src/docs/user/import-mainframe.txt
Lines 228 (patched)
<https://reviews.apache.org/r/62523/#comment295811>

It should be --as-binaryfile instead of ---as-binaryfile



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
Lines 57 (patched)
<https://reviews.apache.org/r/62523/#comment295849>

Unnecessary white space change.



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
Lines 60-61 (patched)
<https://reviews.apache.org/r/62523/#comment295813>

Unused imports.



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 343 (patched)
<https://reviews.apache.org/r/62523/#comment295850>

I think the purpose of this test is to verify that if you create the FTP 
connection, the init commands are executed, right?
In that case it would be better to modify it to use Mockito's verify 
functionality:
```
  @Test
  public void testFtpCommandExecutes() throws IOException {
final String EXPECTED_RESPONSE = "200 OK";
final int EXPECTED_RESPONSE_CODE = 200;
String ftpcmds = "quote SITE RDW,quote SITE RDW READTAPEFORMAT=V";
final int FTP_CMD_COUNT = 2;
when(mockFTPClient.login("user", "pssword")).thenReturn(true);
when(mockFTPClient.logout()).thenReturn(true);
when(mockFTPClient.isConnected()).thenReturn(false);
when(mockFTPClient.getReplyCode()).thenReturn(EXPECTED_RESPONSE_CODE);
when(mockFTPClient.getReplyString()).thenReturn(EXPECTED_RESPONSE);
setupDefaultConfiguration();
conf.set(MainframeConfiguration.MAINFRAME_INPUT_DATASET_TYPE,"g");
conf.set(MainframeConfiguration.MAINFRAME_INPUT_DATASET_NAME,"a.b.c.d");

conf.set(MainframeConfiguration.MAINFRAME_FTP_TRANSFER_MODE,MainframeConfiguration.MAINFRAME_FTP_TRANSFER_MODE_BINARY);
conf.set(MainframeConfiguration.MAINFRAME_FTP_CUSTOM_COMMANDS, ftpcmds);
MainframeFTPClientUtils.setMockFTPClient(mockFTPClient);

MainframeFTPClientUtils.getFTPConnection(conf);

verify(mockFTPClient).sendCommand("quote SITE RDW");
verify(mockFTPClient).sendCommand("quote SITE RDW READTAPEFORMAT=V");
  }
```



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 359-360 (patched)
<https://reviews.apache.org/r/62523/#comment295851>

MainframeFTPClientUtils.getFTPConnection already invokes applyFtpCommands, 
so you don't have to invoke it again.



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 401 (patched)
<https://reviews.apache.org/r/62523/#comment295852>

It would be more straightforward to check if the result array is empty 
here: assertEquals(0, cmds.length)



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 408 (patched)
<https://reviews.apache.org/r/62523/#comment295853>

    It would be more straightforward to check if the result array is empty 
here: assertEquals(0, cmds.length)


- Szabolcs Vasas


On Nov. 29, 2018, 11:46 a.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62523/
> ---
> 
> (Updated Nov. 29, 2018, 11:46 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3237
> https://issues.apache.org/jira/browse/SQOOP-3237
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Added --ftpcmds command to allow comma separated list of FTP commands to send.
> 
> 
> Diffs
> -
> 
>   src/docs/user/import-mainframe.txt 3ecfb7e4 
>   src/java/org/apache/sqoop/SqoopOptions.java f06872f9 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java 
> 9842daa6 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 
> 90dc2ddd 
>   src/java/org/apache/sqoop/tool/MainframeImportTool.java fbc8c3db 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java e7c48a6b 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
>  502e6333 
>   src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 00e57bd0 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java fc6e56d6 
> 
> 
> Diff: https://reviews.apache.org/r/62523/diff/10/
> 
> 
> Testing
> ---
> 
> Unit tests.
> 
> 
> File Attachments
> 
> 
> SQOOP-3237-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/09/26/56041556-e355-4372-83ab-1bcc01680201__SQOOP-3237-1.patch
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas]

Thanks for the question, I think we should stick to the commit format we
had earlier so yes, we should go for squash and push.
The easiest solution could be to download the diff file instead of the
patch (e.g. https://github.com/apache/sqoop/pull/60.diff instead of
https://github.com/apache/sqoop/pull/60.patch) and that does the trick.

The "This closes..." commit message just closes the PR but does not delete
the feature branch, asfgit most probably does not have delete permission
for these branches anyway.


On Fri, Nov 30, 2018 at 11:45 AM Attila Szabó  wrote:

> Hey Szabi,
>
> Thanks for the prompt response!
>
> I've thought the repos are connected back and forth and the close PR is the
> official way. In case of we still stack to the patch file and git apply
> then commit and push approach.
>
> My only question in this case :
> Do we have any agreement on how we commit these PRs. I would vote for
> Squash and push, but of course I'm open for the discussion.
>
> BTW :
> Is "This closes" message deletes the branch automatically?
>
> On front of Github + Jira:
> I'm aware Github has the feature to connect with Jira so I'm pretty sure it
> doable. Also I'm not sure if any ASF project has done it ever, but I'll ask
> around in other communities.
>
> Cheers,
> Attila
>
>
>
> On Nov 30, 2018 11:03 AM, "Szabolcs Vasas"  wrote:
>
> Hi Attila,
>
> I think we won't be able to commit the pull requests on GitHub directly
> because our GitHub repo is just a mirror of the original Apache Sqoop repo
> <https://git-wip-us.apache.org/repos/asf/sqoop.git>.
> However the commit process is still simplified, the ASF GitHub Bot JIRA
> comment contains the patch download link (e.g.
> https://github.com/apache/sqoop/pull/60.patch) and the commit message
> (e.g. This
> closes #60) you need to include to close the pull request. The rest of the
> process remains the same, you need to apply the patch manually and push the
> commit to git-wip-us repo.
>
> Regarding the GitHub UI improvement: I am not sure GitHub provides such a
> feature, do you know an example repository where this is implemented?
>
> Regards,
> Szabolcs
>
>
>
> On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó  wrote:
>
> > Hi Szabi,
> >
> > First of all:
> > Big Kudos for the more mature gradle build! I think this is a great step
> > for the whole project!
> >
> > On the front of PRs:
> > I would only make it official if the user management / authorization
> > handling could be somehow automatically bound to the id.apache.org +
> > project privileges.
> > A good example:
> > Today I reviewed SQOOP-3396 but I would not had been able to merge it
> > because it seems on the Github project I do not have push / merge
> > privileges (regardless that I'm a Sqoop committer and also memeber of the
> > ASF group on github with my user).
> > So if we can somehow bound these things together, and the majority of the
> > ppl would like to use it instead of the Review Board then let it happen!
> > I'm fine with both tools until there's no difference between the Github
> and
> > ASF repos from user management POV.
> >
> > On the top of this:
> > I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> > would be nice if the PRs and the JIRA tickets would be connected
> > automatically on the Github UI as well, and thus the navigation to
> > issues.apache.org would be easier!
> >
> > On the front of the gradle build:
> > I've found a smaller issue with clean+unittest within the same command.
> > I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> > standard) with a solution proposal.
> >
> > My2cents,
> > Attila
> >
> > On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas
>  > >
> > wrote:
> >
> > > Dear Sqoop community,
> > >
> > > We have been working on quite a few exciting build infrastructure
> > > improvements recently, I am sending this email to summarize them.
> > >
> > > *Gradle can now execute all the Sqoop tests in a single JVM*
> > > This improvement makes the Gradle test tasks significantly faster since
> > we
> > > do not have to start up a new JVM for every test class. It also made
> > > possible to introduce fine grained test categories which were essential
> > to
> > > be able to parallelize the test execution in our CI systems. For more
> > > information please refer to COMPILING.txt
> > > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> > >
> > > *Apache Sqoop Jenkins 

Re: Sqoop build infrastructure improvements

[Szabolcs Vasas]

Hi Attila,

I think we won't be able to commit the pull requests on GitHub directly
because our GitHub repo is just a mirror of the original Apache Sqoop repo
<https://git-wip-us.apache.org/repos/asf/sqoop.git>.
However the commit process is still simplified, the ASF GitHub Bot JIRA
comment contains the patch download link (e.g.
https://github.com/apache/sqoop/pull/60.patch) and the commit message
(e.g. This
closes #60) you need to include to close the pull request. The rest of the
process remains the same, you need to apply the patch manually and push the
commit to git-wip-us repo.

Regarding the GitHub UI improvement: I am not sure GitHub provides such a
feature, do you know an example repository where this is implemented?

Regards,
Szabolcs


On Fri, Nov 30, 2018 at 3:21 AM Attila Szabó  wrote:

> Hi Szabi,
>
> First of all:
> Big Kudos for the more mature gradle build! I think this is a great step
> for the whole project!
>
> On the front of PRs:
> I would only make it official if the user management / authorization
> handling could be somehow automatically bound to the id.apache.org +
> project privileges.
> A good example:
> Today I reviewed SQOOP-3396 but I would not had been able to merge it
> because it seems on the Github project I do not have push / merge
> privileges (regardless that I'm a Sqoop committer and also memeber of the
> ASF group on github with my user).
> So if we can somehow bound these things together, and the majority of the
> ppl would like to use it instead of the Review Board then let it happen!
> I'm fine with both tools until there's no difference between the Github and
> ASF repos from user management POV.
>
> On the top of this:
> I'm not sure if it belongs to our table, or the ASF INFRA team, but it
> would be nice if the PRs and the JIRA tickets would be connected
> automatically on the Github UI as well, and thus the navigation to
> issues.apache.org would be easier!
>
> On the front of the gradle build:
> I've found a smaller issue with clean+unittest within the same command.
> I've opened a ticket (SQOOP-3415) and a PR (just the follow the new
> standard) with a solution proposal.
>
> My2cents,
> Attila
>
> On Wed, Nov 28, 2018 at 4:54 PM Szabolcs Vasas  >
> wrote:
>
> > Dear Sqoop community,
> >
> > We have been working on quite a few exciting build infrastructure
> > improvements recently, I am sending this email to summarize them.
> >
> > *Gradle can now execute all the Sqoop tests in a single JVM*
> > This improvement makes the Gradle test tasks significantly faster since
> we
> > do not have to start up a new JVM for every test class. It also made
> > possible to introduce fine grained test categories which were essential
> to
> > be able to parallelize the test execution in our CI systems. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> > *Apache Sqoop Jenkins job
> > <https://builds.apache.org/job/Sqoop-hadoop200/> now builds and tests
> with
> > Gradle*
> > Since our Gradle build became much more stable and faster it made sense
> to
> > reconfigure our Jenkins job to benefit from these improvements. The job
> is
> > faster now (~30 minutes instead of ~40) and it executes all of the tests
> > which can be run without external RDBMS or cloud systems (while the old
> Ant
> > based job executed the unit test suite only).
> >
> > *Travis CI is enabled for Apache Sqoop*
> > The new Travis CI job <https://travis-ci.org/apache/sqoop> now runs for
> > every commit and every pull request on Apache Sqoop GitHub repository and
> > it executes all of the tests except the Oracle third party test cases.
> One
> > of the biggest benefit of Travis CI is that it can be really easily
> > configured for the individual forks as well so contributors get a well
> > configured CI job for their own feature branches for free. For more
> > information please refer to COMPILING.txt
> > <https://github.com/apache/sqoop/blob/trunk/COMPILING.txt>.
> >
> >
> > Since we have a CI job now which integrates very well with GitHub pull
> > requests I suggest deprecating the old Review Board and patch file based
> > contribution process and use pull requests in the future. We had a mail
> > chain about the same proposal last year and it seemed that the community
> > was happy about the idea so I think we can evaluate it for some time and
> if
> > everything goes well we can update our how to contribute wiki.
> >
> > Feel free to reply to this chain with your questions and suggestions on
> the
> > above!
> >
> > Regards,
> > Szabolcs
> >

<http://www.cloudera.com>

Sqoop build infrastructure improvements

[Szabolcs Vasas]

Dear Sqoop community,

We have been working on quite a few exciting build infrastructure
improvements recently, I am sending this email to summarize them.

*Gradle can now execute all the Sqoop tests in a single JVM*
This improvement makes the Gradle test tasks significantly faster since we
do not have to start up a new JVM for every test class. It also made
possible to introduce fine grained test categories which were essential to
be able to parallelize the test execution in our CI systems. For more
information please refer to COMPILING.txt
.

*Apache Sqoop Jenkins job
 now builds and tests with
Gradle*
Since our Gradle build became much more stable and faster it made sense to
reconfigure our Jenkins job to benefit from these improvements. The job is
faster now (~30 minutes instead of ~40) and it executes all of the tests
which can be run without external RDBMS or cloud systems (while the old Ant
based job executed the unit test suite only).

*Travis CI is enabled for Apache Sqoop*
The new Travis CI job  now runs for
every commit and every pull request on Apache Sqoop GitHub repository and
it executes all of the tests except the Oracle third party test cases. One
of the biggest benefit of Travis CI is that it can be really easily
configured for the individual forks as well so contributors get a well
configured CI job for their own feature branches for free. For more
information please refer to COMPILING.txt
.


Since we have a CI job now which integrates very well with GitHub pull
requests I suggest deprecating the old Review Board and patch file based
contribution process and use pull requests in the future. We had a mail
chain about the same proposal last year and it seemed that the community
was happy about the idea so I think we can evaluate it for some time and if
everything goes well we can update our how to contribute wiki.

Feel free to reply to this chain with your questions and suggestions on the
above!

Regards,
Szabolcs

[jira] [Commented] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700691#comment-16700691
 ] 

Szabolcs Vasas commented on SQOOP-3414:
---

I attached 2 txt files to demonstrate the behavior of the new flag during a 
test failure.

To produce a test failure I created an always failing test case(SpecialTest) in 
my working directory which is not present in the patch.

> Introduce a Gradle build parameter to set the ignoreTestFailures of the test 
> tasks
> --
>
> Key: SQOOP-3414
> URL: https://issues.apache.org/jira/browse/SQOOP-3414
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>    Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: test_with_ignoreTestFailures=true.txt, 
> test_without_ignoreTestFailures.txt
>
>
> The 
> [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures]
>  parameter of the Gradle test tasks is set to false which means that if a 
> Gradle test task fails the gradle
> process returns with non-zero. In some CI tools (e.g. Jenkins) this will make 
> the status of the job red and not yellow
> which usually means some more serious issue than a test failure.
> I would like to introduce a parameter to be able set this parameter of the 
> test tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3414:
--
Attachment: test_without_ignoreTestFailures.txt
test_with_ignoreTestFailures=true.txt

> Introduce a Gradle build parameter to set the ignoreTestFailures of the test 
> tasks
> --
>
> Key: SQOOP-3414
> URL: https://issues.apache.org/jira/browse/SQOOP-3414
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>    Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: test_with_ignoreTestFailures=true.txt, 
> test_without_ignoreTestFailures.txt
>
>
> The 
> [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures]
>  parameter of the Gradle test tasks is set to false which means that if a 
> Gradle test task fails the gradle
> process returns with non-zero. In some CI tools (e.g. Jenkins) this will make 
> the status of the job red and not yellow
> which usually means some more serious issue than a test failure.
> I would like to introduce a parameter to be able set this parameter of the 
> test tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3414:
-

 Summary: Introduce a Gradle build parameter to set the 
ignoreTestFailures of the test tasks
 Key: SQOOP-3414
 URL: https://issues.apache.org/jira/browse/SQOOP-3414
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


The 
[ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures]
 parameter of the Gradle test tasks is set to false which means that if a 
Gradle test task fails the gradle
process returns with non-zero. In some CI tools (e.g. Jenkins) this will make 
the status of the job red and not yellow
which usually means some more serious issue than a test failure.
I would like to introduce a parameter to be able set this parameter of the test 
tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3414:
-

Assignee: Szabolcs Vasas

> Introduce a Gradle build parameter to set the ignoreTestFailures of the test 
> tasks
> --
>
> Key: SQOOP-3414
> URL: https://issues.apache.org/jira/browse/SQOOP-3414
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>    Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> The 
> [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures]
>  parameter of the Gradle test tasks is set to false which means that if a 
> Gradle test task fails the gradle
> process returns with non-zero. In some CI tools (e.g. Jenkins) this will make 
> the status of the job red and not yellow
> which usually means some more serious issue than a test failure.
> I would like to introduce a parameter to be able set this parameter of the 
> test tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3412) Travis CI should send notifications of failed builds

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3412.
---
Resolution: Won't Fix

Based on my testing it seems that Travis CI is not able to send emails to 
dev@sqoop.apache.org email so I am resolving this JIRA with Won't Fix.

> Travis CI should send notifications of failed builds
> 
>
> Key: SQOOP-3412
> URL: https://issues.apache.org/jira/browse/SQOOP-3412
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3413) TestMainframeManager does not restore the inner state of AccumuloUtil

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3413:
-

 Summary: TestMainframeManager does not restore the inner state of 
AccumuloUtil
 Key: SQOOP-3413
 URL: https://issues.apache.org/jira/browse/SQOOP-3413
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


org.apache.sqoop.manager.TestMainframeManager#testImportTableNoAccumuloJarPresent
 sets the testingMode field of AccumuloUtil to true but it does not restores it 
so Accumulo tests will fail if they are executed after TestMainframeManager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3413) TestMainframeManager does not restore the inner state of AccumuloUtil

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3413:
-

Assignee: Szabolcs Vasas

> TestMainframeManager does not restore the inner state of AccumuloUtil
> -
>
> Key: SQOOP-3413
> URL: https://issues.apache.org/jira/browse/SQOOP-3413
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> org.apache.sqoop.manager.TestMainframeManager#testImportTableNoAccumuloJarPresent
>  sets the testingMode field of AccumuloUtil to true but it does not restores 
> it so Accumulo tests will fail if they are executed after 
> TestMainframeManager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3412) Travis CI should send notifications of failed builds

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3412:
-

Assignee: Szabolcs Vasas

> Travis CI should send notifications of failed builds
> 
>
> Key: SQOOP-3412
> URL: https://issues.apache.org/jira/browse/SQOOP-3412
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3412) Travis CI should send notifications of failed builds

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3412:
-

 Summary: Travis CI should send notifications of failed builds
 Key: SQOOP-3412
 URL: https://issues.apache.org/jira/browse/SQOOP-3412
 Project: Sqoop
  Issue Type: Test
Reporter: Szabolcs Vasas






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent from the database

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3411:
--
Summary: PostgresMetaConnectIncrementalImportTest fails if metastore tables 
are absent from the database  (was: PostgresMetaConnectIncrementalImportTest 
fails if metastore tables are absent in the database)

> PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent 
> from the database
> ---
>
> Key: SQOOP-3411
> URL: https://issues.apache.org/jira/browse/SQOOP-3411
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3411:
-

 Summary: PostgresMetaConnectIncrementalImportTest fails if 
metastore tables are absent in the database
 Key: SQOOP-3411
 URL: https://issues.apache.org/jira/browse/SQOOP-3411
 Project: Sqoop
  Issue Type: Test
Reporter: Szabolcs Vasas






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3411:
-

Assignee: Szabolcs Vasas

> PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent 
> in the database
> -
>
> Key: SQOOP-3411
> URL: https://issues.apache.org/jira/browse/SQOOP-3411
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3289) Add .travis.yml

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3289:
--
Attachment: SQOOP-3289.patch

> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>    Assignee: Szabolcs Vasas
>Priority: Minor
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3289.patch, SQOOP-3289.patch
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69434: SQOOP-3410: Test S3 import with fs.s3a.security.credential.provider.path

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69434/#review210824
---


Ship it!




Hi Bogi,

Thanks for covering this additional property, ship it!

- Szabolcs Vasas


On Nov. 22, 2018, 4:06 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69434/
> ---
> 
> (Updated Nov. 22, 2018, 4:06 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3410
> https://issues.apache.org/jira/browse/SQOOP-3410
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Based on 
> https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configure_the_hadoop.security.credential.provider.path_property
>  property fs.s3a.security.credential.provider.path can also be used for 
> passing the location of the credential store. This should be also tested and 
> documented.
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt 6ff828c497e0711a2394f768ed5d61ecaf9ec273 
>   src/java/org/apache/sqoop/util/password/CredentialProviderHelper.java 
> 4e79f0ae252969c4a426d1ff69072695eb37b7a6 
>   src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java 
> dca3195b8051048c5c7c2fb3bf30774e9d19eda8 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> e1d7cbda2c65aa59a89715adff52b85fb3730477 
> 
> 
> Diff: https://reviews.apache.org/r/69434/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test
> ./gradlew -Ds3.bucket.url= 
> -Ds3.generator.command= s3Test --tests 
> TestS3ImportWithHadoopCredProvider
> 
> ant clean docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69433: Setting up Travis CI using Gradle test categories

[Szabolcs Vasas]

> On Nov. 22, 2018, 4:41 p.m., Fero Szabo wrote:
> > Hi Szabi,
> > 
> > The whole change looks good to me, haven't spotted any mistakes, though 
> > still need to run tests.
> > 
> > Just some questions to clarify my understanding of the change:
> > 
> > I see the build.xml contains the default values for the connection strings. 
> > 1. How do these get picked up by the docker images?
> > I'm guessing that I can see portforwarding in the yml that's the input for 
> > docker compose, this would answer it.
> > 
> > 2. And how does gradle pick 'em up?
> > I think this is why you've modified the util classes throughout Sqoop. Is 
> > that correct?
> > 
> > So, what are the modifications in the build.xml needed for?

1. Yes, the ports field of the service definition on the yml file defines which 
ports are exposed.
2. Gradle does not need the system properties to be defined in the build.gradle 
file the test will automatically pick up the default value in the 
System.getProperty call. And yes, I modified the default values in all of the 
System.getProperty calls to return the hosts and ports defined in the docker 
compose file.
3. Ant requires the system properties to be defined in build.xml so I had to 
modify the database default host and port values there too.


> On Nov. 22, 2018, 4:41 p.m., Fero Szabo wrote:
> > build.xml
> > Line 193 (original), 197 (patched)
> > <https://reviews.apache.org/r/69433/diff/1/?file=2109980#file2109980line197>
> >
> > I guess localhost could have stayed (just the port had to be added), or 
> > was there a problem with it?

Yes, I remember there was a strange issue with localhost and MySQL direct 
imports, I don't know how but 127.0.0.1 solved it.


- Szabolcs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/#review210809
---


On Nov. 23, 2018, 10:33 a.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69433/
> ---
> 
> (Updated Nov. 23, 2018, 10:33 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3289
> https://issues.apache.org/jira/browse/SQOOP-3289
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The patch includes the following changes:
> - Changed the default DB connection parameters to Docker image defaults so 
> the test tasks can be started without specifying connection parameters
> - Connection parameter settings duplications are removed
> - Most of the JDBC drivers are downloaded from Maven repositories the only 
> exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
> and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
> - Introduced separate test tasks for each databases
> - An Oracle Express Edition Docker image is added to 
> sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
> Oracle EE features can be executed much easier
> - The ports for MySQL and PostgreSQL Docker containers are changed because 
> the default ones were used in the Travis VM already.
> - Introduced OracleEe test category for tests requiring Oracle EE database. 
> These tests won't be executed on Travis. The good news is that only a few 
> tests require Oracle EE
> 
> Documentation is still coming feel free to provide a feedback!
> 
> 
> Diffs
> -
> 
>   .travis.yml PRE-CREATION 
>   COMPILING.txt b399ba825 
>   build.gradle efe980d67 
>   build.xml a0e25191e 
>   gradle.properties 722bc8bb2 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
> PRE-CREATION 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
> fb7800efe 
>   
> src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml
>  b4cf48863 
>   src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
>   
> src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
>  ed949b98f 
>   src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
> 32dfc5eb2 
>   src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
>   src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
> 494c75b08 
>   src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
>   src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
>   src/test/org/apache/sqoop/manager

Re: Review Request 69433: Setting up Travis CI using Gradle test categories

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/
---

(Updated Nov. 23, 2018, 10:33 a.m.)


Review request for Sqoop.


Changes
---

Some minor issues are fixed with Ant and documentation is added.


Bugs: SQOOP-3289
https://issues.apache.org/jira/browse/SQOOP-3289


Repository: sqoop-trunk


Description
---

The patch includes the following changes:
- Changed the default DB connection parameters to Docker image defaults so the 
test tasks can be started without specifying connection parameters
- Connection parameter settings duplications are removed
- Most of the JDBC drivers are downloaded from Maven repositories the only 
exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
- Introduced separate test tasks for each databases
- An Oracle Express Edition Docker image is added to 
sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
Oracle EE features can be executed much easier
- The ports for MySQL and PostgreSQL Docker containers are changed because the 
default ones were used in the Travis VM already.
- Introduced OracleEe test category for tests requiring Oracle EE database. 
These tests won't be executed on Travis. The good news is that only a few tests 
require Oracle EE

Documentation is still coming feel free to provide a feedback!


Diffs (updated)
-

  .travis.yml PRE-CREATION 
  COMPILING.txt b399ba825 
  build.gradle efe980d67 
  build.xml a0e25191e 
  gradle.properties 722bc8bb2 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
PRE-CREATION 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
fb7800efe 
  
src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml 
b4cf48863 
  src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
  
src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
 ed949b98f 
  src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
32dfc5eb2 
  src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
  src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
494c75b08 
  src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
  src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
  src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
  src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
  src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
  src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
34e182f4c 
  src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
  src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
  
src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
 7dd6efcf9 
  src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
1fe264456 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
eb798fa99 
  
src/test/org/apache/sqoop/manager/postgresql/PostgresqlExternalTableImportTest.java
 8c3d2fd90 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlTestUtil.java 
e9705e5da 
  src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java bd12c5566 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerExportTest.java 
ab1e8ff2d 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
3c5bb327e 
  src/test/org/apache/sqoop/metastore/db2/DB2JobToolTest.java 81ef5fce6 
  
src/test/org/apache/sqoop/metastore/db2/DB2MetaConnectIncrementalImportTest.java
 5403908e2 
  src/test/org/apache/sqoop/metastore/db2/DB2SavedJobsTest.java b41eda110 
  src/test/org/apache/sqoop/metastore/postgres/PostgresJobToolTest.java 
59ea151a5 
  
src/test/org/apache/sqoop/metastore/postgres/PostgresMetaConnectIncrementalImportTest.java
 afc6bd232 
  src/test/org/apache/sqoop/metastore/postgres/PostgresSavedJobsTest.java 
9f9e865b9 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/OracleEeTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69433/diff/2/

Changes: https://reviews.apache.org/r/69433/diff/1-2/


Testing
---

The testing was done in my own Sqoop fork with Travis: 
https://travis-ci.org/szvasas/sqoop/builds/458464720


Thanks,

Szabolcs Vasas

[jira] [Updated] (SQOOP-3289) Add .travis.yml

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3289:
--
Issue Type: Sub-task  (was: Task)
Parent: SQOOP-3228

> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3289.patch
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3289) Add .travis.yml

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696056#comment-16696056
 ] 

Szabolcs Vasas commented on SQOOP-3289:
---

Hi All,

This initiative was abandoned for a while, after the Gradle upgrade I have 
managed to create a Travis job which can execute almost all of the third party 
tests with Travis.
Please see my RB request attached: https://reviews.apache.org/r/69433/



> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69433: Setting up Travis CI using Gradle test categories

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/
---

Review request for Sqoop.


Bugs: SQOOP-3289
https://issues.apache.org/jira/browse/SQOOP-3289


Repository: sqoop-trunk


Description
---

The patch includes the following changes:
- Changed the default DB connection parameters to Docker image defaults so the 
test tasks can be started without specifying connection parameters
- Connection parameter settings duplications are removed
- Most of the JDBC drivers are downloaded from Maven repositories the only 
exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
- Introduced separate test tasks for each databases
- An Oracle Express Edition Docker image is added to 
sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
Oracle EE features can be executed much easier
- The ports for MySQL and PostgreSQL Docker containers are changed because the 
default ones were used in the Travis VM already.
- Introduced OracleEe test category for tests requiring Oracle EE database. 
These tests won't be executed on Travis. The good news is that only a few tests 
require Oracle EE

Documentation is still coming feel free to provide a feedback!


Diffs
-

  .travis.yml PRE-CREATION 
  build.gradle efe980d67 
  build.xml a0e25191e 
  gradle.properties 722bc8bb2 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
PRE-CREATION 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
fb7800efe 
  
src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml 
b4cf48863 
  src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
  
src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
 ed949b98f 
  src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
32dfc5eb2 
  src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
  src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
494c75b08 
  src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
  src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
  src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
  src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
  src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
  src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
34e182f4c 
  src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
  src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
  
src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
 7dd6efcf9 
  src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
1fe264456 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
eb798fa99 
  
src/test/org/apache/sqoop/manager/postgresql/PostgresqlExternalTableImportTest.java
 8c3d2fd90 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlTestUtil.java 
e9705e5da 
  src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java bd12c5566 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerExportTest.java 
ab1e8ff2d 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
3c5bb327e 
  src/test/org/apache/sqoop/metastore/db2/DB2JobToolTest.java 81ef5fce6 
  
src/test/org/apache/sqoop/metastore/db2/DB2MetaConnectIncrementalImportTest.java
 5403908e2 
  src/test/org/apache/sqoop/metastore/db2/DB2SavedJobsTest.java b41eda110 
  src/test/org/apache/sqoop/metastore/postgres/PostgresJobToolTest.java 
59ea151a5 
  
src/test/org/apache/sqoop/metastore/postgres/PostgresMetaConnectIncrementalImportTest.java
 afc6bd232 
  src/test/org/apache/sqoop/metastore/postgres/PostgresSavedJobsTest.java 
9f9e865b9 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/OracleEeTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69433/diff/1/


Testing
---

The testing was done in my own Sqoop fork with Travis: 
https://travis-ci.org/szvasas/sqoop/builds/458464720


Thanks,

Szabolcs Vasas

Re: Review Request 69430: SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69430/#review210802
---


Ship it!




Ship It!

- Szabolcs Vasas


On Nov. 22, 2018, 1:37 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69430/
> ---
> 
> (Updated Nov. 22, 2018, 1:37 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3409
> https://issues.apache.org/jira/browse/SQOOP-3409
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. This was 
> caused as the target directory cleanup and name reset happened before the 
> temprootdir cleanup however the temprootdir name depends on the target dir 
> name in the tests.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 2fc606115196a7a2b6088be104e2a421888f8798 
> 
> 
> Diff: https://reviews.apache.org/r/69430/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew s3Test -Ds3.bucket.url= 
> -Ds3.generator.command=, all the used 
> temprootdirs have been cleaned up
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Resolved] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3405.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

Hi [~fero],

Your patch is now committed, thank you for your contribution!



> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!
> We need this to be able to add test categories (so for Travis integration) as 
> well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210795
---


Ship it!




Ship It!

- Szabolcs Vasas


On Nov. 22, 2018, 1:39 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 22, 2018, 1:39 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe1 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebeb 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162d 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd014688 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/7/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Review Request 69429: Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69429/
---

Review request for Sqoop.


Bugs: SQOOP-3408
https://issues.apache.org/jira/browse/SQOOP-3408


Repository: sqoop-trunk


Description
---

Documented forkEvery.default in COMPILING.txt.
Needed to move the definition of the kerberizedTest task below 
tasks.withType(Test) block to preserve forkEvery 1 setting.


Diffs
-

  COMPILING.txt 0383707f689102a3a543d94646cfaaf21710 
  build.gradle 954935daeaaaf45e1b2fd83f74e11f5ed2d58377 


Diff: https://reviews.apache.org/r/69429/diff/1/


Testing
---

./gradlew test : runs the test task with forkEvery=0
./gradlew -DforkEvery.default=5 test : runs the test taks with forkEvery=5

./gradlew kerberizedTest : runs the kerberizedTest task with forkEvery=1
./gradlew -DforkEvery.default=5 kerberizedTest : runs the kerberizedTest task 
with forkEvery=1, so the forkEvery.default parameter does not affect 
kerberizedTest


Thanks,

Szabolcs Vasas

[jira] [Assigned] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3408:
-

Assignee: Szabolcs Vasas

> Introduce a Gradle build parameter to set the default forkEvery value for the 
> tests
> ---
>
> Key: SQOOP-3408
> URL: https://issues.apache.org/jira/browse/SQOOP-3408
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>    Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> The [forkEvery 
> parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
>  of the Gradle test tasks is currently set to 0 which means that all of the 
> tests run in a single JVM (the only exception is the kerberizedTest task 
> which requires a new JVM for every test class).
> The benefit of this setup is that the test tasks finish much faster since the 
> JVM creation is a slow operation. However the Sqoop test framework seems to 
> consume/leak too much memory which can lead to an OutOfMemoryError during the 
> build if there is not enough memory on the machine running the tests.
> The goal of this JIRA is to introduce a new parameter to the Gradle build 
> which can be used to set the default forkEvery parameter and thus prevent the 
> JVM running out of memory.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3408:
-

 Summary: Introduce a Gradle build parameter to set the default 
forkEvery value for the tests
 Key: SQOOP-3408
 URL: https://issues.apache.org/jira/browse/SQOOP-3408
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


The [forkEvery 
parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
 of the Gradle test tasks is currently set to 0 which means that all of the 
tests run in a single JVM (the only exception is the kerberizedTest task which 
requires a new JVM for every test class).
The benefit of this setup is that the test tasks finish much faster since the 
JVM creation is a slow operation. However the Sqoop test framework seems to 
consume/leak too much memory which can lead to an OutOfMemoryError during the 
build if there is not enough memory on the machine running the tests.
The goal of this JIRA is to introduce a new parameter to the Gradle build which 
can be used to set the default forkEvery parameter and thus prevent the JVM 
running out of memory.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210756
---


Ship it!




Hi Feró,

Thank you for improving these tests, unit and third party tests were both 
successful.

- Szabolcs Vasas


On Nov. 21, 2018, 10:20 a.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 21, 2018, 10:20 a.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebe 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd01468 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/5/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69414: Sqoop should not try to execute test category interfaces as tests with Ant

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69414/
---

(Updated Nov. 21, 2018, 12:22 p.m.)


Review request for Sqoop.


Bugs: SQOOP-3406
https://issues.apache.org/jira/browse/SQOOP-3406


Repository: sqoop-trunk


Description
---

When Ant third party test suite is being run Ant tries to execute the test 
category interfaces too because they end with the 'Test' postfix.

These "tests" obviously fail so we need to make sure that Ant does not execute 
them.


Diffs (updated)
-

  build.xml 995a513040f85b6c2043a977a09e93b56913bbed 


Diff: https://reviews.apache.org/r/69414/diff/2/

Changes: https://reviews.apache.org/r/69414/diff/1-2/


Testing
---

ant unit and third party test


Thanks,

Szabolcs Vasas

Review Request 69414: Sqoop should not try to execute test category interfaces as tests with Ant

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69414/
---

Review request for Sqoop.


Bugs: SQOOP-3406
https://issues.apache.org/jira/browse/SQOOP-3406


Repository: sqoop-trunk


Description
---

When Ant third party test suite is being run Ant tries to execute the test 
category interfaces too because they end with the 'Test' postfix.

These "tests" obviously fail so we need to make sure that Ant does not execute 
them.


Diffs
-

  build.xml 995a513040f85b6c2043a977a09e93b56913bbed 


Diff: https://reviews.apache.org/r/69414/diff/1/


Testing
---

ant unit and third party test


Thanks,

Szabolcs Vasas

Review Request 69413: Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69413/
---

Review request for Sqoop.


Bugs: SQOOP-3407
https://issues.apache.org/jira/browse/SQOOP-3407


Repository: sqoop-trunk


Description
---

BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are 
public static fields which get initialized once at the JVM startup and store 
the paths for the test temp and warehouse directories.

The problem is that HBase test cases change the value of the test.build.data 
system property which can cause tests using these static fields to fail.

Since we do not own the code in HBase which changes the system property we need 
to turn these static fields into methods which evaluate the test.build.data 
system property every time they invoked which will make sure that the invoking 
tests will be successful.


Diffs
-

  src/test/org/apache/sqoop/TestIncrementalImport.java 
dbdd05c13e77af514bd996a92f7ebea3a27aedd5 
  src/test/org/apache/sqoop/TestMerge.java 
b283174b8b3df7c16c496795fcbae2f91dd1c375 
  src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java 
9c1e9f9a93323655bc313303bf84d566b551ee00 
  src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java 
df1840b37ce29ffb303b31e1fcbfe4c5842e7c36 
  src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java 
71d6971489e489ae501739fdad5a7409375b6ec1 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
ea7942f62d623895f242e69e77cf9920bbb7e18c 
  src/test/org/apache/sqoop/orm/TestClassWriter.java 
59a8908f13c51b9caca42e8602413ee0b8634b0a 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
e23aad3ee997780e5708e9180550339d834b74d9 


Diff: https://reviews.apache.org/r/69413/diff/1/


Testing
---

Executed unit and third party tests.


Thanks,

Szabolcs Vasas

[jira] [Assigned] (SQOOP-3407) Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3407:
-

Assignee: Szabolcs Vasas

> Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static 
> fields
> 
>
> Key: SQOOP-3407
> URL: https://issues.apache.org/jira/browse/SQOOP-3407
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are 
> public static fields which get initialized once at the JVM startup and store 
> the paths for the test temp and warehouse directories.
> The problem is that HBase test cases change the value of the test.build.data 
> system property which can cause tests using these static fields to fail.
> Since we do not own the code in HBase which changes the system property we 
> need to turn these static fields into methods which evaluate the 
> test.build.data system property every time they invoked which will make sure 
> that the invoking tests will be successful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3407) Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3407:
-

 Summary: Introduce methods instead of TEMP_BASE_DIR and 
LOCAL_WAREHOUSE_DIR static fields
 Key: SQOOP-3407
 URL: https://issues.apache.org/jira/browse/SQOOP-3407
 Project: Sqoop
  Issue Type: Test
Reporter: Szabolcs Vasas


BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are 
public static fields which get initialized once at the JVM startup and store 
the paths for the test temp and warehouse directories.

The problem is that HBase test cases change the value of the test.build.data 
system property which can cause tests using these static fields to fail.

Since we do not own the code in HBase which changes the system property we need 
to turn these static fields into methods which evaluate the test.build.data 
system property every time they invoked which will make sure that the invoking 
tests will be successful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3406) Sqoop should not try to execute test category interfaces as tests with Ant

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3406:
-

 Summary: Sqoop should not try to execute test category interfaces 
as tests with Ant
 Key: SQOOP-3406
 URL: https://issues.apache.org/jira/browse/SQOOP-3406
 Project: Sqoop
  Issue Type: Test
Reporter: Szabolcs Vasas


When Ant third party test suite is being run Ant tries to execute the test 
category interfaces too because they end with the 'Test' postfix.

These "tests" obviously fail so we need to make sure that Ant does not execute 
them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3406) Sqoop should not try to execute test category interfaces as tests with Ant

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3406:
-

Assignee: Szabolcs Vasas

> Sqoop should not try to execute test category interfaces as tests with Ant
> --
>
> Key: SQOOP-3406
> URL: https://issues.apache.org/jira/browse/SQOOP-3406
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
>
> When Ant third party test suite is being run Ant tries to execute the test 
> category interfaces too because they end with the 'Test' postfix.
> These "tests" obviously fail so we need to make sure that Ant does not 
> execute them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3230) Add a build step which starts RDBMS docker containers before the third party tests are executed

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3230.
---
Resolution: Won't Do

Resolving this ticked with 'Won't do' as it seems we will choose a different 
path.

The containers will be started up by a script which will be invoked by the CI 
system.

> Add a build step which starts RDBMS docker containers before the third party 
> tests are executed
> ---
>
> Key: SQOOP-3230
> URL: https://issues.apache.org/jira/browse/SQOOP-3230
> Project: Sqoop
>  Issue Type: Sub-task
>        Reporter: Szabolcs Vasas
>Priority: Major
>
> The task is to enable automatic third party test execution using our build 
> tool.
> Ant should start up all the necessary containers before the tests are 
> executed and shut them down after they are completed.
> The solution should address (at least) the below problems:
> * Find a way to start docker containers from ant (probably a Java library)
> * Random port support: we should make the solution more robust by not relying 
> on the default ports exposed by the containers since they may be occupied on 
> the host running the test cases.
> * Health check logic should be implemented: the framework should make sure 
> that the containers are properly started before it executes the test cases.
> * The solution should not be too tied to Ant since Gradle build will be 
> introduced soon



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/
---

(Updated Nov. 19, 2018, 9:10 a.m.)


Review request for Sqoop.


Bugs: SQOOP-3404
https://issues.apache.org/jira/browse/SQOOP-3404


Repository: sqoop-trunk


Description
---

- All tests are categorized now
- Introduced S3Test category as a subcategory of ThirdPartyTest
- Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
test, s3Test, allTest tasks now
- jacocoTestReport task is fixed to contain the coverage information of the 
kerberizedTest task too. This is needed because the kerberizedTest needs the 
forkEvery parameter to be set to 1 and because of that it has to be a separate 
task which generates separate coverage information too. However it is 
automatically triggered after the test task so the invocation is more 
convenient for the tester.


Diffs (updated)
-

  COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
  build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
  src/test/org/apache/sqoop/manager/TestMainframeManager.java 
c84f05f660c396a06a5031e00abdae77ffbcf2aa 
  
src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
 6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
 5e558717c0d43301ecbf81a37d5ee3fd35756d65 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
 1a6943786834d27f27523f484d76cf678f18cf48 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
 b4cba28c3611400b5c4227a5166b6c91e9152dc4 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
 521a04266e8806321fe7aa6a89c064f369174523 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
  src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
0c3161e5a783446e35f4754124f86715d103ec0b 
  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
5faf59ea80c48fe025294cabd100e7d176032138 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
a4f986423ea299716a29f9d02f7c8453a7f2ba02 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
d271588c5af060bbc3d301a845f45c46d0f6a2ba 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
52d89c775b5f1219471df44d222fd92a59ed408c 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
c9785d816d4a7a5870d74c51a9faa229f6d3818e 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
bba8b74ebe639df26e977abf377f4904144dcfaa 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
114f97cbb8857a7633cae5d030769ac4a90e36aa 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
PRE-CREATION 
  src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
7745f1b07e6d6c457b0164deeace12587ec058d0 


Diff: https://reviews.apache.org/r/69346/diff/4/

Changes: https://reviews.apache.org/r/69346/diff/3-4/


Testing
---

./gradlew unitTest
./gradlew integrationTest
./gradlew kerberizedTest
./gradlew ... s3Test
./gradlew test
./gradlew ... thirdPartyTest
./gradlew allTest


Thanks,

Szabolcs Vasas

Re: Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

> On Nov. 16, 2018, 9:43 p.m., Fero Szabo wrote:
> > COMPILING.txt
> > Line 465 (original), 633 (patched)
> > <https://reviews.apache.org/r/69346/diff/2/?file=2108121#file2108121line639>
> >
> > I believe you require a fully qualified classname here, right? A 
> > concrete example here might come in handy. What do you think?

You don't have to specify a full class name here, and you can use wildcards 
too. I have updated the example and removed -Dtest.single because it is 
deprecated now.


- Szabolcs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/#review210576
---


On Nov. 19, 2018, 9:10 a.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69346/
> ---
> 
> (Updated Nov. 19, 2018, 9:10 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3404
> https://issues.apache.org/jira/browse/SQOOP-3404
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> - All tests are categorized now
> - Introduced S3Test category as a subcategory of ThirdPartyTest
> - Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
> test, s3Test, allTest tasks now
> - jacocoTestReport task is fixed to contain the coverage information of the 
> kerberizedTest task too. This is needed because the kerberizedTest needs the 
> forkEvery parameter to be set to 1 and because of that it has to be a 
> separate task which generates separate coverage information too. However it 
> is automatically triggered after the test task so the invocation is more 
> convenient for the tester.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
>   build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
> c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
>   src/test/org/apache/sqoop/manager/TestMainframeManager.java 
> c84f05f660c396a06a5031e00abdae77ffbcf2aa 
>   
> src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
>  6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
>  5e558717c0d43301ecbf81a37d5ee3fd35756d65 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
>  1a6943786834d27f27523f484d76cf678f18cf48 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
>  b4cba28c3611400b5c4227a5166b6c91e9152dc4 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
>  521a04266e8806321fe7aa6a89c064f369174523 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> 7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
>   src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
> 0c3161e5a783446e35f4754124f86715d103ec0b 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> 3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> 5faf59ea80c48fe025294cabd100e7d176032138 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> a4f986423ea299716a29f9d02f7c8453a7f2ba02 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> d271588c5af060bbc3d301a845f45c46d0f6a2ba 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> 52d89c775b5f1219471df44d222fd92a59ed408c 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> 39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> 597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
> c9785d816d4a7a5870d74c51a9faa229f6d3818e 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> bba8b74ebe639df26e977abf377f4904144dcfaa 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 114f97cbb8857a7633cae5d030769ac4a90e36aa 
>   src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
> 7745f1b07e6d6c457b0164deeace12587ec058d0 
> 
> 
> Diff: https://reviews.apache.org/r/69346/diff/4/
> 
> 
> Testing
> ---
> 
> ./gradlew unitTest
> ./gradlew integrationTest
> ./gradlew kerberizedTest
> ./gradlew ... s3Test
> ./gradlew test
> ./gradlew ... thirdPartyTest
> ./gradlew allTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

> On Nov. 15, 2018, 5:12 p.m., Boglarka Egyed wrote:
> > COMPILING.txt
> > Lines 374-430 (patched)
> > <https://reviews.apache.org/r/69346/diff/2/?file=2108121#file2108121line378>
> >
> > Shouldn't we include S3 related properties here too as those re part of 
> > the third party test suite as well?

Yes, this makes sense.


- Szabolcs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/#review210577
---


On Nov. 15, 2018, 5:45 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69346/
> ---
> 
> (Updated Nov. 15, 2018, 5:45 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3404
> https://issues.apache.org/jira/browse/SQOOP-3404
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> - All tests are categorized now
> - Introduced S3Test category as a subcategory of ThirdPartyTest
> - Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
> test, s3Test, allTest tasks now
> - jacocoTestReport task is fixed to contain the coverage information of the 
> kerberizedTest task too. This is needed because the kerberizedTest needs the 
> forkEvery parameter to be set to 1 and because of that it has to be a 
> separate task which generates separate coverage information too. However it 
> is automatically triggered after the test task so the invocation is more 
> convenient for the tester.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
>   build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
> c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
>   src/test/org/apache/sqoop/manager/TestMainframeManager.java 
> c84f05f660c396a06a5031e00abdae77ffbcf2aa 
>   
> src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
>  6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
>  5e558717c0d43301ecbf81a37d5ee3fd35756d65 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
>  1a6943786834d27f27523f484d76cf678f18cf48 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
>  b4cba28c3611400b5c4227a5166b6c91e9152dc4 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
>  521a04266e8806321fe7aa6a89c064f369174523 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> 7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
>   src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
> 0c3161e5a783446e35f4754124f86715d103ec0b 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> 3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> 5faf59ea80c48fe025294cabd100e7d176032138 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> a4f986423ea299716a29f9d02f7c8453a7f2ba02 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> d271588c5af060bbc3d301a845f45c46d0f6a2ba 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> 52d89c775b5f1219471df44d222fd92a59ed408c 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> 39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> 597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
> c9785d816d4a7a5870d74c51a9faa229f6d3818e 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> bba8b74ebe639df26e977abf377f4904144dcfaa 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 114f97cbb8857a7633cae5d030769ac4a90e36aa 
>   src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
> 7745f1b07e6d6c457b0164deeace12587ec058d0 
> 
> 
> Diff: https://reviews.apache.org/r/69346/diff/3/
> 
> 
> Testing
> ---
> 
> ./gradlew unitTest
> ./gradlew integrationTest
> ./gradlew kerberizedTest
> ./gradlew ... s3Test
> ./gradlew test
> ./gradlew ... thirdPartyTest
> ./gradlew allTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/
---

(Updated Nov. 15, 2018, 5:45 p.m.)


Review request for Sqoop.


Changes
---

Fixed review findings.


Bugs: SQOOP-3404
https://issues.apache.org/jira/browse/SQOOP-3404


Repository: sqoop-trunk


Description
---

- All tests are categorized now
- Introduced S3Test category as a subcategory of ThirdPartyTest
- Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
test, s3Test, allTest tasks now
- jacocoTestReport task is fixed to contain the coverage information of the 
kerberizedTest task too. This is needed because the kerberizedTest needs the 
forkEvery parameter to be set to 1 and because of that it has to be a separate 
task which generates separate coverage information too. However it is 
automatically triggered after the test task so the invocation is more 
convenient for the tester.


Diffs (updated)
-

  COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
  build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
  src/test/org/apache/sqoop/manager/TestMainframeManager.java 
c84f05f660c396a06a5031e00abdae77ffbcf2aa 
  
src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
 6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
 5e558717c0d43301ecbf81a37d5ee3fd35756d65 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
 1a6943786834d27f27523f484d76cf678f18cf48 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
 b4cba28c3611400b5c4227a5166b6c91e9152dc4 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
 521a04266e8806321fe7aa6a89c064f369174523 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
  src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
0c3161e5a783446e35f4754124f86715d103ec0b 
  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
5faf59ea80c48fe025294cabd100e7d176032138 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
a4f986423ea299716a29f9d02f7c8453a7f2ba02 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
d271588c5af060bbc3d301a845f45c46d0f6a2ba 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
52d89c775b5f1219471df44d222fd92a59ed408c 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
c9785d816d4a7a5870d74c51a9faa229f6d3818e 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
bba8b74ebe639df26e977abf377f4904144dcfaa 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
114f97cbb8857a7633cae5d030769ac4a90e36aa 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
PRE-CREATION 
  src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
7745f1b07e6d6c457b0164deeace12587ec058d0 


Diff: https://reviews.apache.org/r/69346/diff/3/

Changes: https://reviews.apache.org/r/69346/diff/2-3/


Testing
---

./gradlew unitTest
./gradlew integrationTest
./gradlew kerberizedTest
./gradlew ... s3Test
./gradlew test
./gradlew ... thirdPartyTest
./gradlew allTest


Thanks,

Szabolcs Vasas

Re: Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/
---

(Updated Nov. 15, 2018, 1:40 p.m.)


Review request for Sqoop.


Changes
---

Updated COMPILING.txt to document the test tasks we have now with Gradle.
It was outdated and not properly formatted in a few places so I updated those 
parts as well.
Please check the formatting by running asciidoc COMPILING.txt and opening the 
output HTML.


Bugs: SQOOP-3404
https://issues.apache.org/jira/browse/SQOOP-3404


Repository: sqoop-trunk


Description
---

- All tests are categorized now
- Introduced S3Test category as a subcategory of ThirdPartyTest
- Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
test, s3Test, allTest tasks now
- jacocoTestReport task is fixed to contain the coverage information of the 
kerberizedTest task too. This is needed because the kerberizedTest needs the 
forkEvery parameter to be set to 1 and because of that it has to be a separate 
task which generates separate coverage information too. However it is 
automatically triggered after the test task so the invocation is more 
convenient for the tester.


Diffs (updated)
-

  COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
  build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
  src/test/org/apache/sqoop/manager/TestMainframeManager.java 
c84f05f660c396a06a5031e00abdae77ffbcf2aa 
  
src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
 6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
 5e558717c0d43301ecbf81a37d5ee3fd35756d65 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
 1a6943786834d27f27523f484d76cf678f18cf48 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
 b4cba28c3611400b5c4227a5166b6c91e9152dc4 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
 521a04266e8806321fe7aa6a89c064f369174523 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
  src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
0c3161e5a783446e35f4754124f86715d103ec0b 
  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
5faf59ea80c48fe025294cabd100e7d176032138 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
a4f986423ea299716a29f9d02f7c8453a7f2ba02 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
d271588c5af060bbc3d301a845f45c46d0f6a2ba 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
52d89c775b5f1219471df44d222fd92a59ed408c 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
c9785d816d4a7a5870d74c51a9faa229f6d3818e 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
bba8b74ebe639df26e977abf377f4904144dcfaa 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
114f97cbb8857a7633cae5d030769ac4a90e36aa 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
PRE-CREATION 
  src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
7745f1b07e6d6c457b0164deeace12587ec058d0 


Diff: https://reviews.apache.org/r/69346/diff/2/

Changes: https://reviews.apache.org/r/69346/diff/1-2/


Testing
---

./gradlew unitTest
./gradlew integrationTest
./gradlew kerberizedTest
./gradlew ... s3Test
./gradlew test
./gradlew ... thirdPartyTest
./gradlew allTest


Thanks,

Szabolcs Vasas

Review Request 69346: Categorize all tests in the project

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/
---

Review request for Sqoop.


Bugs: SQOOP-3404
https://issues.apache.org/jira/browse/SQOOP-3404


Repository: sqoop-trunk


Description
---

- All tests are categorized now
- Introduced S3Test category as a subcategory of ThirdPartyTest
- Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
test, s3Test, allTest tasks now
- jacocoTestReport task is fixed to contain the coverage information of the 
kerberizedTest task too. This is needed because the kerberizedTest needs the 
forkEvery parameter to be set to 1 and because of that it has to be a separate 
task which generates separate coverage information too. However it is 
automatically triggered after the test task so the invocation is more 
convenient for the tester.


Diffs
-

  build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
  src/test/org/apache/sqoop/manager/TestMainframeManager.java 
c84f05f660c396a06a5031e00abdae77ffbcf2aa 
  
src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
 6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
 5e558717c0d43301ecbf81a37d5ee3fd35756d65 
  
src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
 1a6943786834d27f27523f484d76cf678f18cf48 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
 b4cba28c3611400b5c4227a5166b6c91e9152dc4 
  
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
 521a04266e8806321fe7aa6a89c064f369174523 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
  src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
0c3161e5a783446e35f4754124f86715d103ec0b 
  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
5faf59ea80c48fe025294cabd100e7d176032138 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
a4f986423ea299716a29f9d02f7c8453a7f2ba02 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
d271588c5af060bbc3d301a845f45c46d0f6a2ba 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
52d89c775b5f1219471df44d222fd92a59ed408c 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
c9785d816d4a7a5870d74c51a9faa229f6d3818e 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
bba8b74ebe639df26e977abf377f4904144dcfaa 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
114f97cbb8857a7633cae5d030769ac4a90e36aa 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
PRE-CREATION 
  src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
7745f1b07e6d6c457b0164deeace12587ec058d0 


Diff: https://reviews.apache.org/r/69346/diff/1/


Testing
---

./gradlew unitTest
./gradlew integrationTest
./gradlew kerberizedTest
./gradlew ... s3Test
./gradlew test
./gradlew ... thirdPartyTest
./gradlew allTest


Thanks,

Szabolcs Vasas

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Issue Type: Sub-task  (was: Test)
Parent: SQOOP-3228

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Sub-task
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> SQOOP-3104 has introduced test categories but while it was under review many 
> other patches with new test cases were committed.
> The task is to make sure that all of the new tests are properly categorized 
> and the test tasks are cleaned up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Description: 
SQOOP-3104 has introduced test categories but while it was under review many 
other patches with new test cases were committed.

The task is to make sure that all of the new tests are properly categorized and 
the test tasks are cleaned up.

  was:
We are currently unsing test naming conventions to differentiate between 
ManualTests, Unit tests and 3rd party tests.

Instead of this, we should use junit4 categories, so it's clear by looking at a 
test where it fits. This would also remove the reliance on the test class name.



> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> SQOOP-3104 has introduced test categories but while it was under review many 
> other patches with new test cases were committed.
> The task is to make sure that all of the new tests are properly categorized 
> and the test tasks are cleaned up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3404:
-

Assignee: Szabolcs Vasas  (was: Nguyen Truong)

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>        Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests.
> Instead of this, we should use junit4 categories, so it's clear by looking at 
> a test where it fits. This would also remove the reliance on the test class 
> name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3404:
-

 Summary: Categorize all tests in the project
 Key: SQOOP-3404
 URL: https://issues.apache.org/jira/browse/SQOOP-3404
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Szabolcs Vasas
Assignee: Nguyen Truong
 Fix For: 3.0.0


We are currently unsing test naming conventions to differentiate between 
ManualTests, Unit tests and 3rd party tests.

Instead of this, we should use junit4 categories, so it's clear by looking at a 
test where it fits. This would also remove the reliance on the test class name.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Issue Type: Test  (was: Sub-task)
Parent: (was: SQOOP-3228)

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>    Reporter: Szabolcs Vasas
>Assignee: Nguyen Truong
>Priority: Major
> Fix For: 3.0.0
>
>
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests.
> Instead of this, we should use junit4 categories, so it's clear by looking at 
> a test where it fits. This would also remove the reliance on the test class 
> name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68541: SQOOP-3104: Create test categories instead of test suites and naming conventions

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68541/#review210541
---


Ship it!




Hi Nguyen,

Thank you for updating your patch, I ran the unit and third party tests and 
everything is green.

- Szabolcs Vasas


On Nov. 13, 2018, 6:18 a.m., Nguyen Truong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68541/
> ---
> 
> (Updated Nov. 13, 2018, 6:18 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3104
> https://issues.apache.org/jira/browse/SQOOP-3104
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests. Instead of that, I implemented 
> junit categories which will allow us to have more categories in the future. 
> This would also remove the reliance on the test class name.
> 
> Test categories skeleton:
>   SqoopTest _ UnitTest
>   |__ IntegrationTest
>   |__ ManualTest
> 
>   ThirdPartyTest _ CubridTest
>|__ Db2Test
>|__ MainFrameTest
>|__ MysqlTest
>|__ NetezzaTest
>|__ OracleTest
>|__ PostgresqlTest
>|__ SqlServerTest
> 
>   KerberizedTest
> 
> Categories explanation:
> * SqoopTest: Group of the big categories, including:
> - UnitTest: It tests one class only with its dependencies mocked or 
> if the dependency
> is lightweight we can keep it. It must not start a minicluster or an 
> hsqldb database.
> It does not need JCDB drivers.
> - IntegrationTest: It usually tests a whole scenario. It may start up 
> miniclusters,
> hsqldb and connect to external resources like RDBMSs.
> - ManualTest: This should be a deprecated category which should not 
> be used in the future.
> It only exists to mark the currently existing manual tests.
> * ThirdPartyTest: An orthogonal hierarchy for tests that need a JDBC 
> driver and/or a docker
> container/external RDBMS instance to run. Subcategories express what kind 
> of external
> resource the test needs. E.g: OracleTest needs an Oracle RDBMS and Oracle 
> driver on the classpath
> * KerberizedTest: Test that needs Kerberos, which needs to be run on a 
> separate JVM.
> 
> Opinions are very welcomed. Thanks!
> 
> 
> Diffs
> -
> 
>   build.gradle 2014b5cf5 
>   src/test/org/apache/sqoop/TestConnFactory.java fb6c94059 
>   src/test/org/apache/sqoop/TestIncrementalImport.java 29c477954 
>   src/test/org/apache/sqoop/TestSqoopOptions.java e55682edf 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java 631eeff5e 
>   src/test/org/apache/sqoop/authentication/TestKerberosAuthenticator.java 
> f5700ce65 
>   src/test/org/apache/sqoop/db/TestDriverManagerJdbcConnectionFactory.java 
> 244831672 
>   
> src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
>  d3e3fb23e 
>   src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java c4caafba5 
>   src/test/org/apache/sqoop/hbase/HBaseKerberizedConnectivityTest.java 
> 3bfb39178 
>   src/test/org/apache/sqoop/hbase/HBaseTestCase.java 94b71b61c 
>   src/test/org/apache/sqoop/hbase/HBaseUtilTest.java c6a808c33 
>   src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java e78a535f4 
>   src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java ba05cabbb 
>   
> src/test/org/apache/sqoop/hive/HiveServer2ConnectionFactoryInitializerTest.java
>  4d2cb2f88 
>   src/test/org/apache/sqoop/hive/TestHiveClientFactory.java a3c2dc939 
>   src/test/org/apache/sqoop/hive/TestHiveMiniCluster.java 419f888c0 
>   src/test/org/apache/sqoop/hive/TestHiveServer2Client.java 02617295e 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java 65f079467 
>   src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java 410724f37 
>   src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java 
> 276e9eaa4 
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java 626ad22f6 
>   src/test/org/apache/sqoop/hive/TestTableDefWriterForExternalTable.java 
> f1768ee76 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   src/test/org/apache/sqoop/io/TestCode

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review210519
---


Ship it!




Hi Feró,

Thank you for fixing all the findings, I have ran the unit and third party 
tests with the latest patch and all of them passed.
Ship it!

- Szabolcs Vasas


On Nov. 12, 2018, 4:33 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Nov. 12, 2018, 4:33 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
>  e82154309 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b9 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967f 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912a 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c0 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c171 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41bd 
>   
> src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
>  f137b56b7 
>   
> src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/4/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 68989: [SQOOP-3387 Include Column-Remarks

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68989/#review210023
---



Hi Tomas,

Thank you for submitting this improvement, please find my findings below.

Apart from those you should add your test coverage to your code, as for example 
currently nothing proves that the queries in 
org.apache.sqoop.manager.SqlManager#getColumnRemarks and 
org.apache.sqoop.manager.SqlManager#getTableRemark work.
Also it is a bit confusing that this logic was added to SqlManager but 
according to the description only Oracle is supported.
I think we should either move this implementation down to OracleManager or test 
it for all the databases.

Regards,
Szabolcs


src/java/org/apache/sqoop/manager/ConnManager.java
Lines 353 (patched)
<https://reviews.apache.org/r/68989/#comment294660>

nit: TreeMap is redundant, you can use the diamond operator 
here: TreeMap<>



src/java/org/apache/sqoop/manager/SqlManager.java
Lines 303 (patched)
<https://reviews.apache.org/r/68989/#comment294662>

The log message does not seem to fit this method.



src/java/org/apache/sqoop/manager/oracle/OraOopConnManager.java
Lines 537 (patched)
<https://reviews.apache.org/r/68989/#comment295061>

This method seems to be very similar to getColumnTypes, can we somehow 
eliminate the duplication?



src/java/org/apache/sqoop/manager/oracle/OraOopConnManager.java
Lines 568 (patched)
<https://reviews.apache.org/r/68989/#comment294663>

This is an unnecessary override.



src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
Line 110 (original), 113 (patched)
<https://reviews.apache.org/r/68989/#comment295064>

I am not sure if changing the doc field here is safe as some people could 
rely or would like to have the original table name kept. 
What if you just append the table remark to the doc if exists?



src/java/org/apache/sqoop/orm/ClassWriter.java
Lines 1893 (patched)
<https://reviews.apache.org/r/68989/#comment295062>

IOException is never thrown here



src/java/org/apache/sqoop/orm/ClassWriter.java
Lines 1897 (patched)
<https://reviews.apache.org/r/68989/#comment295063>

IOException is never thrown here.



src/java/org/apache/sqoop/orm/ClassWriter.java
Lines 1899 (patched)
<https://reviews.apache.org/r/68989/#comment294658>

nit: please remove the trailing whitespace from this line


- Szabolcs Vasas


On Oct. 11, 2018, 7:23 a.m., Tomas Sebastian Hätälä wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68989/
> ---
> 
> (Updated Oct. 11, 2018, 7:23 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3387
> https://issues.apache.org/jira/browse/SQOOP-3387
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> In most RDBMS it is possible to enter comments/ remarks for table and view 
> columns. That way a user can obtain additional information regarding the data 
> and how to use it.
> 
> With the avro file format it would be possible to store this information in 
> the schema file using the "doc"-tag. At the moment this is, however, left 
> blanc.
> 
> This patch includes table and column remarks for Oracle DB and Avro
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/manager/ConnManager.java 4c1e8f5 
>   src/java/org/apache/sqoop/manager/SqlManager.java d82332a 
>   src/java/org/apache/sqoop/manager/oracle/OraOopConnManager.java 95eaacf 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9 
>   src/java/org/apache/sqoop/orm/ClassWriter.java 46d0698 
> 
> 
> Diff: https://reviews.apache.org/r/68989/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tomas Sebastian Hätälä
> 
>

[jira] [Resolved] (SQOOP-2943) Make sqoop able to import to Parquet file format in case of HDFS encryption zones are turned on

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-2943.
---
Resolution: Duplicate

Resolving this as this issue is solved by 
https://issues.apache.org/jira/browse/SQOOP-3313

> Make sqoop able to import to Parquet file format in case of HDFS encryption 
> zones are turned on
> ---
>
> Key: SQOOP-2943
> URL: https://issues.apache.org/jira/browse/SQOOP-2943
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Attila Szabo
>Priority: Major
> Fix For: 1.5.0
>
>
> If HDFS encryption zones are turned on, and the user tries to import into 
> Parquet format, where the target location is in a very different encryption 
> zone, than the zone of the /tmp/ location (typical use case for that 
> encrypted hive warehouse directory), even the mapper jobs are executed 
> successfully, and the partial results stored on the temp storage correctly, 
> the MergeOutputMapper class of Kite SDK dies with an HDFS related exception 
> ("can't be moved into an encryption zone").
> The problem does not appear in case of clear text output formats.
> Please make Sqoop able to solve this scenario as well!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2949) SQL Syntax error when split-by column is of character type and min or max value has single quote inside it

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670306#comment-16670306
 ] 

Szabolcs Vasas commented on SQOOP-2949:
---

Hi [~gireeshp], [~fero],

I have committed both the fix and the new tests, thank you for your efforts on 
this!

I am resolving the JIRAs, [~fero] please resolve the RB request.

 

Thanks,

Szabolcs

> SQL Syntax error when split-by column is of character type and min or max 
> value has single quote inside it
> --
>
> Key: SQOOP-2949
> URL: https://issues.apache.org/jira/browse/SQOOP-2949
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Sqoop 1.4.6
> Run on Hadoop 2.6.0
> On Ubuntu
>Reporter: Gireesh Puthumana
>Assignee: Gireesh Puthumana
>Priority: Major
> Fix For: 3.0.0
>
>
> Did a sqoop import from mysql table "emp", with split-by column "ename", 
> which is a varchar(100) type.
> +Used below command:+
> sqoop import --connect jdbc:mysql://localhost/testdb --username root 
> --password * --table emp --m 2 --target-dir /sqoopTest/5 --split-by ename;
> +Ename has following records:+
> | ename   |
> | gireesh |
> | aavesh  |
> | shiva'  |
> | jamir   |
> | balu|
> | santosh |
> | sameer  |
> Min value is "aavesh" and max value is "shiva'" (please note the single quote 
> inside max value).
> When run, it tried to execute below query in mapper 2 and failed:
> SELECT `ename`, `eid`, `deptid` FROM `emp` AS `emp` WHERE ( `ename` >= 
> 'jd聯聭聪G耀' ) AND ( `ename` <= 'shiva'' )
> +Stack trace:+
> {quote}
> 2016-06-05 16:54:06,749 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error 
> in your SQL syntax; check the manual that corresponds to your MySQL server 
> version for the right syntax to use near ''shiva'' )' at line 1
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>   at com.mysql.jdbc.Util.getInstance(Util.java:387)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:942)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>   at 
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69199: Create tests for SQOOP-2949, quote escaping in split-by

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69199/#review210215
---




src/test/org/apache/sqoop/importjob/SplitByImportTest.java
Lines 21 (patched)
<https://reviews.apache.org/r/69199/#comment294834>

Unused import.



src/test/org/apache/sqoop/importjob/SplitByImportTest.java
Lines 29 (patched)
<https://reviews.apache.org/r/69199/#comment294835>

Unused import.



src/test/org/apache/sqoop/importjob/SplitByImportTest.java
Lines 137 (patched)
<https://reviews.apache.org/r/69199/#comment294837>

Since the num-mappers is 1 the split-by logic does not get invoked, can you 
please set it to 2?



src/test/org/apache/sqoop/importjob/SplitByImportTest.java
Lines 152 (patched)
<https://reviews.apache.org/r/69199/#comment294836>

Since the readAll* methods of ParquetReader close the reader this method 
could be simplified to something like this:

private void verifyParquetFile() {
ParquetReader reader = new ParquetReader(new Path(getWarehouseDir() + 
"/" + getTableName()), getConf());
assertEquals(asList(configuration.getExpectedResultsForParquet()), 
reader.readAllInCsv());
  }



src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
Lines 60 (patched)
<https://reviews.apache.org/r/69199/#comment294833>

I suggest simplifying this method by using streams:

return data.stream()
.map(element -> StringUtils.join(element, SEPARATOR))
.toArray(String[]::new);



src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
Lines 61 (patched)
<https://reviews.apache.org/r/69199/#comment294832>

We should not hardcode the size of the String[] here.



src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
Lines 63 (patched)
<https://reviews.apache.org/r/69199/#comment294831>

This variable is unused.


- Szabolcs Vasas


On Oct. 30, 2018, 4:26 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69199/
> ---
> 
> (Updated Oct. 30, 2018, 4:26 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3400
> https://issues.apache.org/jira/browse/SQOOP-3400
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Integration tests fro SQOOP-2949.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java 22bbfe68 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java fe6ba831 
> 
> 
> Diff: https://reviews.apache.org/r/69199/diff/3/
> 
> 
> Testing
> ---
> 
> This is the testing part for a fix that lacked testing. 
> gradle test and gradle 3rdpartytests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review210083
---




src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java
Lines 292 (patched)
<https://reviews.apache.org/r/69060/#comment294724>

Since the readAll* methods of ParquetReader close the reader this method 
could be simplified to something like this:

private void verifyParquetFile() {
ParquetReader reader = new ParquetReader(new Path(getWarehouseDir() + 
"/" + getTableName()), getConf());
assertEquals(asList(configuration.getExpectedResultsForParquet()), 
reader.readAllInCsv());
  }


- Szabolcs Vasas


On Oct. 24, 2018, 12:25 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Oct. 24, 2018, 12:25 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c17 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41b 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/2/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Szabolcs Vasas]

> On Oct. 24, 2018, 2:31 p.m., Szabolcs Vasas wrote:
> > Hi Feró,
> > 
> > Thank you for submitting this improvement!
> > I have left some comments, see them below.
> > Apart from that I think we need to test explicitly that if the 
> > sqoop.parquet.logical_types.decimal.enable flag is true then the Parquet 
> > file contains a decimal value and otherwise it contains a string value.
> > 
> > NumericTypesImportTest asserts on string values so it is not able to verify 
> > this, most of the tests passed even if I commented out the content of the 
> > addEnableParquetDecimal method.
> 
> Fero Szabo wrote:
> I'll look into this one.
> 
> I'm thinking that using the org.apache.sqoop.util.ParquetReader#readAll 
> method could help (since it returns GenericRecords), though I'm not sure. 
> I'll somehow need to actually turn off the conversion and check for the 
> bytes. Any suggestions?

Yes, I would run a Sqoop job with and without conversion switched on and check 
the schema of the GenericRecord returned by 
org.apache.sqoop.util.ParquetReader#readAll


> On Oct. 24, 2018, 2:31 p.m., Szabolcs Vasas wrote:
> > src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
> > Lines 56 (patched)
> > <https://reviews.apache.org/r/69060/diff/2/?file=2099870#file2099870line56>
> >
> > Are we sure that adding the logical type conversion only here is enough?
> > In case of Avro it is also added in 
> > org.apache.sqoop.mapreduce.AvroOutputFormat#getRecordWriter which gets 
> > invoked in every mapper so I assume that we have to add the conversion in 
> > every mapper in case of Parquet files too.
> 
> Fero Szabo wrote:
> My understanding is that this method is invoked in every mapper. (it's 
> doc suggest this as well: "Called once at the beginning of the task.")
> Where else would put this statement?

Sorry, you are right, this should be enough here.


- Szabolcs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review209954
---


On Oct. 24, 2018, 12:25 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Oct. 24, 2018, 12:25 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  

Re: Review Request 69164: SQOOP-3395: Document Hadoop CredentialProvider usage in case of import into S3

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69164/#review210037
---


Ship it!




Nice improvement for our docs again, let's ship it!

- Szabolcs Vasas


On Oct. 25, 2018, 1:10 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69164/
> ---
> 
> (Updated Oct. 25, 2018, 1:10 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3395
> https://issues.apache.org/jira/browse/SQOOP-3395
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Document Hadoop CredentialProvider usage in case of import into S3
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt 52ab6ac07203494922db7a7aaa991c2ea1fc52c8 
> 
> 
> Diff: https://reviews.apache.org/r/69164/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-3232) Remove Sqoop dependency on deprecated HBase APIs

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662406#comment-16662406
 ] 

Szabolcs Vasas commented on SQOOP-3232:
---

Hi [~szewinho],

Yes, this improvement will be included in the next Sqoop release, I am not sure 
about the version number yet, it is going to be 1.5.0 or 3.0.0.

We are still working on some non-trivial library upgrades 
(https://issues.apache.org/jira/browse/SQOOP-3305) most probably we will have 
to wait for an HBase release that supports Hadoop 3.x properly and the 
community has to find the time for preparing the release so there is no ETA yet.

 

> Remove Sqoop dependency on deprecated HBase APIs
> 
>
> Key: SQOOP-3232
> URL: https://issues.apache.org/jira/browse/SQOOP-3232
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3232-2.patch, SQOOP-3232.patch
>
>
> Sqoop currently depends on pre HBase 1.0 APIs that have been deprecated and 
> will be removed in the HBase 2.0 release.
> The task is to remove the dependency on these old APIs to make sure that the 
> upgrade to a newer HBase version will be easier in the future.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69066: SQOOP-3390: Document S3Guard usage with Sqoop

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69066/#review209965
---


Ship it!




Ship It!

- Szabolcs Vasas


On Oct. 17, 2018, 4:49 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69066/
> ---
> 
> (Updated Oct. 17, 2018, 4:49 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3390
> https://issues.apache.org/jira/browse/SQOOP-3390
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Document Hadoop's S3Guard usage with Sqoop to overcome Amazon S3's eventual 
> consistency.
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt c54b26bc5ef71f8cd7d18ce6eb98a296dbffed92 
> 
> 
> Diff: https://reviews.apache.org/r/69066/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review209954
---



Hi Feró,

Thank you for submitting this improvement!
I have left some comments, see them below.
Apart from that I think we need to test explicitly that if the 
sqoop.parquet.logical_types.decimal.enable flag is true then the Parquet file 
contains a decimal value and otherwise it contains a string value.

NumericTypesImportTest asserts on string values so it is not able to verify 
this, most of the tests passed even if I commented out the content of the 
addEnableParquetDecimal method.


src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
Lines 115-119 (patched)
<https://reviews.apache.org/r/69060/#comment294605>

Is it possible to move this to 
org.apache.sqoop.mapreduce.parquet.hadoop.HadoopParquetImportJobConfigurator#configureMapper?
That would be consistent with the way we configure the Parquet imports but 
I am not sure the effect would remain the same.



src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
Lines 56 (patched)
<https://reviews.apache.org/r/69060/#comment294606>

Are we sure that adding the logical type conversion only here is enough?
In case of Avro it is also added in 
org.apache.sqoop.mapreduce.AvroOutputFormat#getRecordWriter which gets invoked 
in every mapper so I assume that we have to add the conversion in every mapper 
in case of Parquet files too.


- Szabolcs Vasas


On Oct. 24, 2018, 12:25 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Oct. 24, 2018, 12:25 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c17 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41b 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/2/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 68064: SQOOP-3355 Document SQOOP-1905 DB2 --schema option

[Szabolcs Vasas]

> On Oct. 24, 2018, 12:21 p.m., Szabolcs Vasas wrote:
> > src/docs/user/connectors.txt
> > Lines 41 (patched)
> > <https://reviews.apache.org/r/68064/diff/1/?file=2064419#file2064419line41>
> >
> > Are you sure import-all-tables supports --schema properly?
> > By looking at the code of this command it seems to me that Sqoop uses 
> > the list-tables command to get the table names it wants to import but then 
> > it invokes the import command in a loop to import all the tables. So I 
> > think that even if the list-tables shows a table the import command will 
> > not be able to pick it up.
> 
> Fero Szabo wrote:
> Hi Szabi,
> 
> Yes, I've tested this in July, when I created the review. Now this 
> reminds me to write down everything. :)
> 
> Anyway, I've tested again, and this works as expected. What I did was: 
> fire up a cluster, created a couple of tables in a DB2 instance and ran the 
> command. 2 different schemas, all the data form all of the tables are in 
> hdfs. I'd say this is proper behavior.
> 
> Now I'm not sure why/how this feature wasn't implemented for a (simple) 
> import, but it just wasn't. The error message suggests that sqoop tries to 
> use the default schema, i.e. DB2INST1. We can open a Jira if there isn't one.
> 
> I'm dropping this issue.

Thank you for double checking this!


- Szabolcs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68064/#review209952
---


On July 26, 2018, 2:58 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68064/
> ---
> 
> (Updated July 26, 2018, 2:58 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3355
> https://issues.apache.org/jira/browse/SQOOP-3355
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Adding documentation for a previously implemented feature. This one is quite 
> simple.
> 
> 
> Diffs
> -
> 
>   src/docs/user/connectors.txt 59e3e00b 
> 
> 
> Diff: https://reviews.apache.org/r/68064/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs, 
> + unit and 3rd party tests, though these shouldn't be affected.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 68064: SQOOP-3355 Document SQOOP-1905 DB2 --schema option

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68064/#review209960
---


Ship it!




Ship It!

- Szabolcs Vasas


On July 26, 2018, 2:58 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68064/
> ---
> 
> (Updated July 26, 2018, 2:58 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3355
> https://issues.apache.org/jira/browse/SQOOP-3355
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Adding documentation for a previously implemented feature. This one is quite 
> simple.
> 
> 
> Diffs
> -
> 
>   src/docs/user/connectors.txt 59e3e00b 
> 
> 
> Diff: https://reviews.apache.org/r/68064/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs, 
> + unit and 3rd party tests, though these shouldn't be affected.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Review Request 69141: Tests using HiveMiniCluster can be unstable on some platforms

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69141/
---

Review request for Sqoop.


Bugs: SQOOP-3398
https://issues.apache.org/jira/browse/SQOOP-3398


Repository: sqoop-trunk


Description
---

Since the last Hive upgrade TestHiveMiniCluster fails on some platforms because 
and older version of the ASM library is picked up.

The task is to exclude the older ASM library in ivy and gradle to make sure the 
test passes on all platforms.


Diffs
-

  build.gradle 2340bce7519a46b203a287a4b5160c62e0c09509 
  ivy.xml 6805fc329d44bcc0707e7cab67f3749a42e6f769 


Diff: https://reviews.apache.org/r/69141/diff/1/


Testing
---

Executed unit and third party tests with both ant and gradle.


Thanks,

Szabolcs Vasas

Re: Review Request 68064: SQOOP-3355 Document SQOOP-1905 DB2 --schema option

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68064/#review209952
---




src/docs/user/connectors.txt
Lines 41 (patched)
<https://reviews.apache.org/r/68064/#comment294603>

Are you sure import-all-tables supports --schema properly?
By looking at the code of this command it seems to me that Sqoop uses the 
list-tables command to get the table names it wants to import but then it 
invokes the import command in a loop to import all the tables. So I think that 
even if the list-tables shows a table the import command will not be able to 
pick it up.


- Szabolcs Vasas


On July 26, 2018, 2:58 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68064/
> ---
> 
> (Updated July 26, 2018, 2:58 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3355
> https://issues.apache.org/jira/browse/SQOOP-3355
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Adding documentation for a previously implemented feature. This one is quite 
> simple.
> 
> 
> Diffs
> -
> 
>   src/docs/user/connectors.txt 59e3e00b 
> 
> 
> Diff: https://reviews.apache.org/r/68064/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs, 
> + unit and 3rd party tests, though these shouldn't be affected.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3399:
--
Attachment: SQOOP-3399.patch

> TestS3ImportWithHadoopCredProvider fails if credential generator command is 
> not provided
> 
>
> Key: SQOOP-3399
> URL: https://issues.apache.org/jira/browse/SQOOP-3399
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3399.patch
>
>
> BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw 
> NullPointerException when the credential generator command is not provided 
> since it fails the test with Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69139: TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69139/
---

Review request for Sqoop.


Bugs: SQOOP-3399
https://issues.apache.org/jira/browse/SQOOP-3399


Repository: sqoop-trunk


Description
---

BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw 
NullPointerException when the credential generator command is not provided 
since it fails the test with Gradle.


Diffs
-

  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
e03eb64ef 


Diff: https://reviews.apache.org/r/69139/diff/1/


Testing
---

Executed the test with both and and gradle, with and without S3 credential 
generator provided.


Thanks,

Szabolcs Vasas

[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3399:
--
Description: BeforeClass method of TestS3ImportWithHadoopCredProvider 
should not throw NullPointerException when the credential generator command is 
not provided since it fails the test with Gradle.  (was: The task is to fix )

> TestS3ImportWithHadoopCredProvider fails if credential generator command is 
> not provided
> 
>
> Key: SQOOP-3399
> URL: https://issues.apache.org/jira/browse/SQOOP-3399
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>    Assignee: Szabolcs Vasas
>Priority: Major
>
> BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw 
> NullPointerException when the credential generator command is not provided 
> since it fails the test with Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3399:
--
Summary: TestS3ImportWithHadoopCredProvider fails if credential generator 
command is not provided  (was: TestS3ImportWithHadoopCredProvider fails if 
credential generator coma is not provided)

> TestS3ImportWithHadoopCredProvider fails if credential generator command is 
> not provided
> 
>
> Key: SQOOP-3399
> URL: https://issues.apache.org/jira/browse/SQOOP-3399
> Project: Sqoop
>  Issue Type: Test
>        Reporter: Szabolcs Vasas
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)