[jira] [Commented] (SQOOP-3428) Fix the CI

[Fero Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952792#comment-16952792
 ] 

Fero Szabo commented on SQOOP-3428:
---

[~fokko], I've added this Jira to the commit description. If you've had a 
different one in mind, we can amend the commit! Otherwise we should close this 
one, as the CI appears to be fixed. :) 

Thanks again!

> Fix the CI
> --
>
> Key: SQOOP-3428
> URL: https://issues.apache.org/jira/browse/SQOOP-3428
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the CI is broken because the Oracle 11 XE Dockerimage isn't 
> available anymore.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (SQOOP-3451) Importing FLOAT from Oracle to Hive results in INTEGER

[Fero Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937767#comment-16937767
 ] 

Fero Szabo edited comment on SQOOP-3451 at 9/25/19 2:17 PM:


Hi [~dionusos],

Yeah, I think you are right and Oracle is a pain in this regard to work with.

I've had the same issue when developing the fixed point decimal support for 
Avro and Parquet, namely that a column defined as NUMBER (without precision and 
scale), comes back with invalid metadata from the database. (I believe 
something like -127 as scale, though please double check this). And under the 
hood, I suspect Oracle is using NUMBER, again, to store the type Float. 

In my case, the only missing thing was a proper scale to be able to pad a 
BigDecimal within sqoop. So, I created a flag to allow this to the user in 
SQOOP-2976. Not sure what to do in your case, as it's neither of those files, 
(it's orc, if I'm seeing this correctly). In any case, I believe you'll need to 
watch out for these "special" values for scale and precision returned by Oracle 
and implement a logic that maps these to proper values.

I used user input for this via properties. Seemed the best at the time, for 
that particular case. I'm not sure if my approach is the right one for you as 
well, though certainly an option.

So, TL;DR:

Track down where the Hive schema gets created and debug whether you can 
identify a Float coming from Oracle based on the precision and scale. You might 
want to check other number types, too.

Hope this helps!

 

(edited a mistake)


was (Author: fero):
Hi [~dionusos],

Yeah, I think you are right and Oracle is a pain in this regard to work with.

I've had the same issue when developing the fixed point number support for Avro 
and Parquet, namely that a column defined as NUMBER (without precision and 
scale), comes back with invalid metadata from the database. (I believe 
something like -127 as scale, though please double check this). And under the 
hood, I suspect Oracle is using NUMBER, again, to store the type Float. 

In my case, the only missing thing was a proper scale to be able to pad a 
BigDecimal within sqoop. So, I created a flag to allow this to the user in 
SQOOP-2976. Not sure what to do in your case, as it's neither of those files, 
(it's orc, if I'm seeing this correctly). In any case, I believe you'll need to 
watch out for these "special" values for scale and precision returned by Oracle 
and implement a logic that maps these to proper values.

I used user input for this via properties. Seemed the best at the time, for 
that particular case. I'm not sure if my approach is the right one for you as 
well, though certainly an option.

So, TL;DR:

Track down where the Hive schema gets created and debug whether you can 
identify a Float coming from Oracle based on the precision and scale. You might 
want to check other number types, too.

Hope this helps!

 

(edited a mistake)

> Importing FLOAT from Oracle to Hive results in INTEGER
> --
>
> Key: SQOOP-3451
> URL: https://issues.apache.org/jira/browse/SQOOP-3451
> Project: Sqoop
>  Issue Type: Bug
>  Components: codegen, connectors/oracle, hive-integration
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Major
>
> We ran into an issue where there is a table created in Oracle 11g:
> {noformat}
> create table floattest (column1 float(30), column2 number(30,-127), column3 
> number(30));
> {noformat}
> We want to import date from Oracle to Hive:
> {noformat}
> sqoop import -D 
> mapred.child.java.opts='-Djava.security.egd=file:/dev/../dev/urandom' 
> -Dmapreduce.job.queuename=default --connect 
> "jdbc:oracle:thin:@DBHOST:1521/xe" --username sqoop --password sqoop --table 
> floattest --hcatalog-database default --hcatalog-table floattest 
> --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza 
> "stored as orc" -m 1 --columns COLUMN1,COLUMN2,COLUMN3 --verbose
> {noformat}
> In Sqoop logs we see the following:
> {noformat}
> 19/09/24 13:51:45 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM floattest t WHERE 1=0
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN1 of type [2, 
> 30, -127]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN2 of type [2, 
> 30, -84]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN3 of type [2, 
> 30, 0]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column names 
> projected : [COLUMN1, COLUMN2, COLUMN3]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column name - info 
> map :
> COLUMN3 : [Type 

[jira] [Comment Edited] (SQOOP-3451) Importing FLOAT from Oracle to Hive results in INTEGER

[Fero Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937767#comment-16937767
 ] 

Fero Szabo edited comment on SQOOP-3451 at 9/25/19 2:17 PM:


Hi [~dionusos],

Yeah, I think you are right and Oracle is a pain in this regard to work with.

I've had the same issue when developing the fixed point number support for Avro 
and Parquet, namely that a column defined as NUMBER (without precision and 
scale), comes back with invalid metadata from the database. (I believe 
something like -127 as scale, though please double check this). And under the 
hood, I suspect Oracle is using NUMBER, again, to store the type Float. 

In my case, the only missing thing was a proper scale to be able to pad a 
BigDecimal within sqoop. So, I created a flag to allow this to the user in 
SQOOP-2976. Not sure what to do in your case, as it's neither of those files, 
(it's orc, if I'm seeing this correctly). In any case, I believe you'll need to 
watch out for these "special" values for scale and precision returned by Oracle 
and implement a logic that maps these to proper values.

I used user input for this via properties. Seemed the best at the time, for 
that particular case. I'm not sure if my approach is the right one for you as 
well, though certainly an option.

So, TL;DR:

Track down where the Hive schema gets created and debug whether you can 
identify a Float coming from Oracle based on the precision and scale. You might 
want to check other number types, too.

Hope this helps!

 

(edited a mistake)


was (Author: fero):
Hi [~dionusos],

Yeah, I think you are right and Oracle is a pain in this regard to work with.

I've had the same issue when developing the floating point number support for 
Avro and Parquet, namely that a column defined as NUMBER (without precision and 
scale), comes back with invalid metadata from the database. (I believe 
something like -127 as scale, though please double check this). And under the 
hood, I suspect Oracle is using NUMBER, again, to store the type Float. 

In my case, the only missing thing was a proper scale to be able to pad a 
BigDecimal within sqoop. So, I created a flag to allow this to the user in 
SQOOP-2976. Not sure what to do in your case, as it's neither of those files, 
(it's orc, if I'm seeing this correctly). In any case, I believe you'll need to 
watch out for these "special" values for scale and precision returned by Oracle 
and implement a logic that maps these to proper values.

I used user input for this via properties. Seemed the best at the time, for 
that particular case. I'm not sure if my approach is the right one for you as 
well, though certainly an option.

So, TL;DR:

Track down where the Hive schema gets created and debug whether you can 
identify a Float coming from Oracle based on the precision and scale. You might 
want to check other number types, too.

Hope this helps!

> Importing FLOAT from Oracle to Hive results in INTEGER
> --
>
> Key: SQOOP-3451
> URL: https://issues.apache.org/jira/browse/SQOOP-3451
> Project: Sqoop
>  Issue Type: Bug
>  Components: codegen, connectors/oracle, hive-integration
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Major
>
> We ran into an issue where there is a table created in Oracle 11g:
> {noformat}
> create table floattest (column1 float(30), column2 number(30,-127), column3 
> number(30));
> {noformat}
> We want to import date from Oracle to Hive:
> {noformat}
> sqoop import -D 
> mapred.child.java.opts='-Djava.security.egd=file:/dev/../dev/urandom' 
> -Dmapreduce.job.queuename=default --connect 
> "jdbc:oracle:thin:@DBHOST:1521/xe" --username sqoop --password sqoop --table 
> floattest --hcatalog-database default --hcatalog-table floattest 
> --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza 
> "stored as orc" -m 1 --columns COLUMN1,COLUMN2,COLUMN3 --verbose
> {noformat}
> In Sqoop logs we see the following:
> {noformat}
> 19/09/24 13:51:45 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM floattest t WHERE 1=0
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN1 of type [2, 
> 30, -127]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN2 of type [2, 
> 30, -84]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN3 of type [2, 
> 30, 0]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column names 
> projected : [COLUMN1, COLUMN2, COLUMN3]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column name - info 
> map :
> COLUMN3 : [Type : 2,P

[jira] [Commented] (SQOOP-3451) Importing FLOAT from Oracle to Hive results in INTEGER

[Fero Szabo (Jira)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937767#comment-16937767
 ] 

Fero Szabo commented on SQOOP-3451:
---

Hi [~dionusos],

Yeah, I think you are right and Oracle is a pain in this regard to work with.

I've had the same issue when developing the floating point number support for 
Avro and Parquet, namely that a column defined as NUMBER (without precision and 
scale), comes back with invalid metadata from the database. (I believe 
something like -127 as scale, though please double check this). And under the 
hood, I suspect Oracle is using NUMBER, again, to store the type Float. 

In my case, the only missing thing was a proper scale to be able to pad a 
BigDecimal within sqoop. So, I created a flag to allow this to the user in 
SQOOP-2976. Not sure what to do in your case, as it's neither of those files, 
(it's orc, if I'm seeing this correctly). In any case, I believe you'll need to 
watch out for these "special" values for scale and precision returned by Oracle 
and implement a logic that maps these to proper values.

I used user input for this via properties. Seemed the best at the time, for 
that particular case. I'm not sure if my approach is the right one for you as 
well, though certainly an option.

So, TL;DR:

Track down where the Hive schema gets created and debug whether you can 
identify a Float coming from Oracle based on the precision and scale. You might 
want to check other number types, too.

Hope this helps!

> Importing FLOAT from Oracle to Hive results in INTEGER
> --
>
> Key: SQOOP-3451
> URL: https://issues.apache.org/jira/browse/SQOOP-3451
> Project: Sqoop
>  Issue Type: Bug
>  Components: codegen, connectors/oracle, hive-integration
>Affects Versions: 1.4.7
>Reporter: Denes Bodo
>Priority: Major
>
> We ran into an issue where there is a table created in Oracle 11g:
> {noformat}
> create table floattest (column1 float(30), column2 number(30,-127), column3 
> number(30));
> {noformat}
> We want to import date from Oracle to Hive:
> {noformat}
> sqoop import -D 
> mapred.child.java.opts='-Djava.security.egd=file:/dev/../dev/urandom' 
> -Dmapreduce.job.queuename=default --connect 
> "jdbc:oracle:thin:@DBHOST:1521/xe" --username sqoop --password sqoop --table 
> floattest --hcatalog-database default --hcatalog-table floattest 
> --create-hcatalog-table --hcatalog-external-table --hcatalog-storage-stanza 
> "stored as orc" -m 1 --columns COLUMN1,COLUMN2,COLUMN3 --verbose
> {noformat}
> In Sqoop logs we see the following:
> {noformat}
> 19/09/24 13:51:45 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM floattest t WHERE 1=0
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN1 of type [2, 
> 30, -127]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN2 of type [2, 
> 30, -84]
> 19/09/24 13:51:45 DEBUG manager.SqlManager: Found column COLUMN3 of type [2, 
> 30, 0]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column names 
> projected : [COLUMN1, COLUMN2, COLUMN3]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Database column name - info 
> map :
> COLUMN3 : [Type : 2,Precision : 30,Scale : 0]
> COLUMN2 : [Type : 2,Precision : 30,Scale : -84]
> COLUMN1 : [Type : 2,Precision : 30,Scale : -127]
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: Creating HCatalog table 
> default.floattest for import
> 19/09/24 13:51:45 INFO hcat.SqoopHCatUtilities: HCatalog Create table 
> statement:
> create external table `default`.`floattest` (
> `column1` decimal(30),
> `column2` decimal(30),
> `column3` decimal(30))
> stored as orc
> {noformat}
> From this output we can see that Oracle states about column1 has Type=2 which 
> is NUMERIC (regarding to 
> https://docs.oracle.com/javase/7/docs/api/constant-values.html#java.sql.Types.FLOAT).
>  Sqoop translates NUMERIC to DECIMAL 
> (https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java#L1050L1107).
>  Due to Oracle uses {{scale=-127}} to sign about a NUMERIC that it is a FLOAT 
> instead of stating {{Type=6}}, Sqoop creates integers (decimal with 0 scale) 
> from NUMBER.
> I think it is the fault of Oracle as it does not use Java Type=6 to sign type 
> of a float. What do you think?
> 
> Thank you for the details and investigation to [~mbalakrishnan] and Andrew 
> Miller



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (SQOOP-3442) Sqoop Java 11 support

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3442:
-

 Summary: Sqoop Java 11 support
 Key: SQOOP-3442
 URL: https://issues.apache.org/jira/browse/SQOOP-3442
 Project: Sqoop
  Issue Type: Improvement
Reporter: Fero Szabo


In order for Sqoop to support Java 11, we'll have to bump the dependencies for 
the following:
 * Hadoop
 * Hive
 * HBase

This will be a major undertaking.

SQOOP-3441 took care of the necessary code changes.

We (the community) should also consider dropping ant support entirely, as 
hammering out the dependency upgrade in two build systems just doesn't make 
sense.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3441) Prepare Sqoop for Java 11 support

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3441:
-

 Summary: Prepare Sqoop for Java 11 support
 Key: SQOOP-3441
 URL: https://issues.apache.org/jira/browse/SQOOP-3441
 Project: Sqoop
  Issue Type: Improvement
Reporter: Fero Szabo
Assignee: Fero Szabo


A couple of code changes will be required in order for Sqoop to work with 
Java11 and we'll also have to bump a couple of dependencies and the gradle 
version. 

I'm not sure what's required for ant, that is to be figured out in a separate 
Jira, if we keep the ant build.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3430) Fix broken CI

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3430:
--
Description: I think both ant and gradle are enough  
!/jira/images/icons/emoticons/smile.png|width=16,height=16!  (was: Currently, 
the CI is broken because the Oracle 11 XE Dockerimage isn't available anymore.)

> Fix broken CI
> -
>
> Key: SQOOP-3430
> URL: https://issues.apache.org/jira/browse/SQOOP-3430
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
> I think both ant and gradle are enough  
> !/jira/images/icons/emoticons/smile.png|width=16,height=16!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3430) Remove the old maven pom

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789439#comment-16789439
 ] 

Fero Szabo commented on SQOOP-3430:
---

[~Fokko],

Could you please link your PR to this jira? (not sure if a new one needs to be 
opened)



I see no reason to keep the old maven pom.

> Remove the old maven pom
> 
>
> Key: SQOOP-3430
> URL: https://issues.apache.org/jira/browse/SQOOP-3430
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
> I think both ant and gradle are enough :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3428) Fix the CI

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3428:
--
Summary: Fix the CI  (was: Remove the old Maven pom)

> Fix the CI
> --
>
> Key: SQOOP-3428
> URL: https://issues.apache.org/jira/browse/SQOOP-3428
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I think both ant and gradle are enough :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3428) Fix the CI

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3428:
--
Description: Currently, the CI is broken because the Oracle 11 XE 
Dockerimage isn't available anymore.  (was: I think both ant and gradle are 
enough :-))

> Fix the CI
> --
>
> Key: SQOOP-3428
> URL: https://issues.apache.org/jira/browse/SQOOP-3428
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the CI is broken because the Oracle 11 XE Dockerimage isn't 
> available anymore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3430) Remove the old maven pom

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3430:
--
Description: I think both ant and gradle are enough :)  (was: I think both 
ant and gradle are enough  
!/jira/images/icons/emoticons/smile.png|width=16,height=16!)

> Remove the old maven pom
> 
>
> Key: SQOOP-3430
> URL: https://issues.apache.org/jira/browse/SQOOP-3430
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>
> I think both ant and gradle are enough :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3428) Fix the CI

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789433#comment-16789433
 ] 

Fero Szabo commented on SQOOP-3428:
---

Hi [~Fokko],

I changed the Summary and removed the description to match that of the pull 
request. Apparently there was a mismatch I didn't spot in time (the Jira 
referenced in the pull request was this one, instead of SQOOP-3430, that you've 
opened for fixing the CI). 

Anyway, this one is resolved and I am repurposing SQOOP-3430 for removing the 
old pom.

> Fix the CI
> --
>
> Key: SQOOP-3428
> URL: https://issues.apache.org/jira/browse/SQOOP-3428
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I think both ant and gradle are enough :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3428) Fix the CI

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo reassigned SQOOP-3428:
-

Assignee: Fokko Driesprong

> Fix the CI
> --
>
> Key: SQOOP-3428
> URL: https://issues.apache.org/jira/browse/SQOOP-3428
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I think both ant and gradle are enough :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 62523: SQOOP-3237: Mainframe FTP transfer option to insert custom FTP commands prior to transfer

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62523/#review211186
---


Fix it, then Ship it!




Hi Chris,

I also had a look at this one. Nice, clean patch, good job!

I just had a few comments on formatting, no biggies. We should probably 
automate this formatting stuff in the future somehow. Also the removal of 
unused imports. 

Best Regards,
Fero


src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java
Lines 78 (patched)
<https://reviews.apache.org/r/62523/#comment296113>

You could also use isNotBlank next time :)

That's slightly easier on my eyes.



src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java
Lines 281 (patched)
<https://reviews.apache.org/r/62523/#comment296115>

Missing space. Should read like this:
LOG.info("Issuing command: " + ftpCommand);

I'm not sure if we have a formatting guidline like this, but we probably 
should if we don't.



src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java
Lines 285 (patched)
<https://reviews.apache.org/r/62523/#comment296116>

Also missing space around the + operators.

LOG.info("ReplyCode: " + res + " ReplyString: " + result);



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
Lines 59 (patched)
<https://reviews.apache.org/r/62523/#comment296114>

This new line is unnecessary here.



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 21-22 (patched)
<https://reviews.apache.org/r/62523/#comment296117>

Unused import



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 34 (patched)
<https://reviews.apache.org/r/62523/#comment296118>

Unused import.


- Fero Szabo


On Dec. 10, 2018, 5:45 a.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62523/
> ---
> 
> (Updated Dec. 10, 2018, 5:45 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3237
> https://issues.apache.org/jira/browse/SQOOP-3237
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Added --ftpcmds command to allow comma separated list of FTP commands to send.
> 
> 
> Diffs
> -
> 
>   src/docs/user/import-mainframe.txt 3ecfb7e4 
>   src/java/org/apache/sqoop/SqoopOptions.java f06872f9 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java 
> 9842daa6 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 
> 90dc2ddd 
>   src/java/org/apache/sqoop/tool/MainframeImportTool.java fbc8c3db 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java e7c48a6b 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
>  502e6333 
>   src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 00e57bd0 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java fc6e56d6 
> 
> 
> Diff: https://reviews.apache.org/r/62523/diff/11/
> 
> 
> Testing
> ---
> 
> Unit tests.
> 
> 
> File Attachments
> 
> 
> SQOOP-3237-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/09/26/56041556-e355-4372-83ab-1bcc01680201__SQOOP-3237-1.patch
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Re: Mandatory relocation of Sqoop git repository to gitbox

[Fero Szabo]

Hi Szabi,

+1 from my side to get this done in the initial phase!

Best Regards,
Fero

On Mon, Dec 10, 2018 at 2:29 PM Szabolcs Vasas  wrote:

> Hi All,
>
> According to this
> <
> http://mail-archives.apache.org/mod_mbox/incubator-general/201812.mbox/%3c6edbcae6-4eb9-6f61-beac-4198fd750...@apache.org%3E
> >
> email
> the git-wip-us server, which hosts Apache Sqoop git repository too, is
> going to be decommissioned soon and all the projects are going to be
> migrated to gitbox.
> For the detailed description of the planned change please refer to the
> email linked, but the bottom line is that after the migration we are going
> to be able to merge pull requests on the GitHub UI as well which will
> greatly simplify our commit process.
>
> This relocation is mandatory however we have the option execute it in the
> initial phase which would be great in my opinion because we could start
> enjoying the benefits very soon.
>
> Please reply to this chain with your opinion because we need a consensus to
> be able to start the migration in the initial phase.
>
> Thanks and regards,
> Szabolcs
>


-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

[jira] [Created] (SQOOP-3418) Document decimal support in Hive external import into parquet files

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3418:
-

 Summary: Document decimal support in Hive external import into 
parquet files
 Key: SQOOP-3418
 URL: https://issues.apache.org/jira/browse/SQOOP-3418
 Project: Sqoop
  Issue Type: Task
Reporter: Fero Szabo
Assignee: Fero Szabo


Remember to note the limitations in Hive i.e. the max scale and precision is 38 
and how it behaves in edge cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3416) Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable in the gradle build

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo resolved SQOOP-3416.
---
Resolution: Won't Fix

This became obsolete, as the variable was deleted.

> Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable 
> in the gradle build
> ---
>
> Key: SQOOP-3416
> URL: https://issues.apache.org/jira/browse/SQOOP-3416
> Project: Sqoop
>  Issue Type: Improvement
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Minor
>
> Since the sqoopThirdPartyLib doesn't have a default value, if one runs the 
> Oracle tests, one always have to specify the sqoop.thirdparty.lib.dir system 
> variable.
> With this change, we just have to move the downloaded oracle driver to 
> /var/lib/dir and avoid some typing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3417) Execute Oracle XE tests on Travis CI

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo resolved SQOOP-3417.
---
Resolution: Fixed

> Execute Oracle XE tests on Travis CI
> 
>
> Key: SQOOP-3417
> URL: https://issues.apache.org/jira/browse/SQOOP-3417
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
>
> The task is to enable the Travis CI to execute Oracle XE tests too 
> automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3417) Execute Oracle XE tests on Travis CI

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708772#comment-16708772
 ] 

Fero Szabo commented on SQOOP-3417:
---

Hi [~vasas],

Your change is now committed. Thank you for your contribution, good catch!

> Execute Oracle XE tests on Travis CI
> 
>
> Key: SQOOP-3417
> URL: https://issues.apache.org/jira/browse/SQOOP-3417
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
>
> The task is to enable the Travis CI to execute Oracle XE tests too 
> automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3416) Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable in the gradle build

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3416:
-

 Summary: Give the default value of /var/lib/sqoop to the 
sqoopThirdPartyLib variable in the gradle build
 Key: SQOOP-3416
 URL: https://issues.apache.org/jira/browse/SQOOP-3416
 Project: Sqoop
  Issue Type: Improvement
Reporter: Fero Szabo
Assignee: Fero Szabo


Since the sqoopThirdPartyLib doesn't have a default value, if one runs the 
Oracle tests, one always have to specify the sqoop.thirdparty.lib.dir system 
variable.

With this change, we just have to move the downloaded oracle driver to 
/var/lib/dir and avoid some typing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Next release

[Fero Szabo]

Hi Attila, Ferenc

I would be happy to drive the next release!

As for scoping, there is another thread that discusses this topic with the
subject: Release to support Hadoop 3

There is only one item missing from that original scope: upgrading Hadoop
to 3.x. We'll need to make this upgrade in order to support the newest
versions of Hive, HBase and Accumulo. I remember that we were waiting for a
new release from HBase. HBase version 2.1.1 was released on 2018/10/31, but
I'm not sure if / when / how the Hadoop upgrade will move forward and if
this new HBase has what we need exactly.

Best Regards,
Fero

On Tue, Nov 27, 2018 at 5:55 PM Attila Szabó  wrote:

> Hi Ferenc,
>
> I did not have any specific plan for the end of the year, but as I did the
> very last release I would be happy to help anyone who would like to drive
> it ( or if no one volunteers I might own it, but IMHO from community POV it
> would be better if someone else would execute it this time).
>
> On the scoping front :
> AFAIR there are a few tickets which seems to be abandoned for a while and
> targeted to this release. A grooming there would be a great start for the
> scoping.
>
> My2cents,
> Attila
>
> On Tue, Nov 27, 2018, 2:10 PM Ferenc Szabo 
> > Hi sqoop developers,
> >
> > Do you have a plan for the next release?
> >
> > Regards
> > Ferenc
> >
>


-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

Re: Review Request 69434: SQOOP-3410: Test S3 import with fs.s3a.security.credential.provider.path

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69434/#review210832
---


Ship it!




Lgtm

- Fero Szabo


On Nov. 22, 2018, 4:06 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69434/
> ---
> 
> (Updated Nov. 22, 2018, 4:06 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3410
> https://issues.apache.org/jira/browse/SQOOP-3410
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Based on 
> https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configure_the_hadoop.security.credential.provider.path_property
>  property fs.s3a.security.credential.provider.path can also be used for 
> passing the location of the credential store. This should be also tested and 
> documented.
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt 6ff828c497e0711a2394f768ed5d61ecaf9ec273 
>   src/java/org/apache/sqoop/util/password/CredentialProviderHelper.java 
> 4e79f0ae252969c4a426d1ff69072695eb37b7a6 
>   src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java 
> dca3195b8051048c5c7c2fb3bf30774e9d19eda8 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> e1d7cbda2c65aa59a89715adff52b85fb3730477 
> 
> 
> Diff: https://reviews.apache.org/r/69434/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test
> ./gradlew -Ds3.bucket.url= 
> -Ds3.generator.command= s3Test --tests 
> TestS3ImportWithHadoopCredProvider
> 
> ant clean docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69433: Setting up Travis CI using Gradle test categories

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/#review210825
---


Ship it!




Hi Szabi,

not an issue, just an idea.

In any case, tests are passing, and it looks good.


COMPILING.txt
Lines 413 (patched)
<https://reviews.apache.org/r/69433/#comment295583>

I think it would make sense to try to pick up the oracle driver from 
/var/lib/sqoop. What do you think?


- Fero Szabo


On Nov. 23, 2018, 10:33 a.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69433/
> ---
> 
> (Updated Nov. 23, 2018, 10:33 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3289
> https://issues.apache.org/jira/browse/SQOOP-3289
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The patch includes the following changes:
> - Changed the default DB connection parameters to Docker image defaults so 
> the test tasks can be started without specifying connection parameters
> - Connection parameter settings duplications are removed
> - Most of the JDBC drivers are downloaded from Maven repositories the only 
> exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
> and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
> - Introduced separate test tasks for each databases
> - An Oracle Express Edition Docker image is added to 
> sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
> Oracle EE features can be executed much easier
> - The ports for MySQL and PostgreSQL Docker containers are changed because 
> the default ones were used in the Travis VM already.
> - Introduced OracleEe test category for tests requiring Oracle EE database. 
> These tests won't be executed on Travis. The good news is that only a few 
> tests require Oracle EE
> 
> Documentation is still coming feel free to provide a feedback!
> 
> 
> Diffs
> -
> 
>   .travis.yml PRE-CREATION 
>   COMPILING.txt b399ba825 
>   build.gradle efe980d67 
>   build.xml a0e25191e 
>   gradle.properties 722bc8bb2 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
> PRE-CREATION 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
> fb7800efe 
>   
> src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml
>  b4cf48863 
>   src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
>   
> src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
>  ed949b98f 
>   src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
> 32dfc5eb2 
>   src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
>   src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
> 494c75b08 
>   src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
>   src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
>   src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
>   src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
> 34e182f4c 
>   src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
>   src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
>   
> src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
>  7dd6efcf9 
>   
> src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
> 1fe264456 
>   src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
> eb798fa99 
>   
> src/test/org/apache/sqoop/manager/postgresql/PostgresqlExternalTableImportTest.java
>  8c3d2fd90 
>   src/test/org/apache/sqoop/manager/postgresql/PostgresqlTestUtil.java 
> e9705e5da 
>   src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java bd12c5566 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerExportTest.java 
> ab1e8ff2d 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
> 3c5bb327e 
>   src/test/org/apache/sqoop/metastore/db2/DB2JobToolTest.java 81ef5fce6 
>   
> src/test/org/apache/sqoop/metastore/db2/DB2MetaConnectIncrementalImportTest.java
>  5403908e2 
>   src/test/org/apache/sqoop/metastore/db2/DB2SavedJobsTest.java b41eda110 
>   src/test/org

Re: Review Request 69433: Setting up Travis CI using Gradle test categories

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/#review210809
---



Hi Szabi,

The whole change looks good to me, haven't spotted any mistakes, though still 
need to run tests.

Just some questions to clarify my understanding of the change:

I see the build.xml contains the default values for the connection strings. 
1. How do these get picked up by the docker images?
I'm guessing that I can see portforwarding in the yml that's the input for 
docker compose, this would answer it.

2. And how does gradle pick 'em up?
I think this is why you've modified the util classes throughout Sqoop. Is that 
correct?

So, what are the modifications in the build.xml needed for?


build.xml
Line 193 (original), 197 (patched)
<https://reviews.apache.org/r/69433/#comment295568>

I guess localhost could have stayed (just the port had to be added), or was 
there a problem with it?



src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java
Line 56 (original), 56 (patched)
<https://reviews.apache.org/r/69433/#comment295569>

uppercase / lowercase typo :)

mysqlDbNAme > mysqlDbName


- Fero Szabo


On Nov. 22, 2018, 3:59 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69433/
> ---
> 
> (Updated Nov. 22, 2018, 3:59 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3289
> https://issues.apache.org/jira/browse/SQOOP-3289
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The patch includes the following changes:
> - Changed the default DB connection parameters to Docker image defaults so 
> the test tasks can be started without specifying connection parameters
> - Connection parameter settings duplications are removed
> - Most of the JDBC drivers are downloaded from Maven repositories the only 
> exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
> and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
> - Introduced separate test tasks for each databases
> - An Oracle Express Edition Docker image is added to 
> sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
> Oracle EE features can be executed much easier
> - The ports for MySQL and PostgreSQL Docker containers are changed because 
> the default ones were used in the Travis VM already.
> - Introduced OracleEe test category for tests requiring Oracle EE database. 
> These tests won't be executed on Travis. The good news is that only a few 
> tests require Oracle EE
> 
> Documentation is still coming feel free to provide a feedback!
> 
> 
> Diffs
> -
> 
>   .travis.yml PRE-CREATION 
>   build.gradle efe980d67 
>   build.xml a0e25191e 
>   gradle.properties 722bc8bb2 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
> PRE-CREATION 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
> fb7800efe 
>   
> src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml
>  b4cf48863 
>   src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
>   
> src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
>  ed949b98f 
>   src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
> 32dfc5eb2 
>   src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
>   src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
> 494c75b08 
>   src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
>   src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
>   src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
>   src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
> 34e182f4c 
>   src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
>   src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
>   
> src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
>  7dd6efcf9 
>   
> src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
> 1fe264456 
>   src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
> eb798fa99 
>   
> src/test/org/apache/sqoop/manager/postgresql

Re: Review Request 69430: SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69430/#review210803
---


Ship it!




Ship It!

- Fero Szabo


On Nov. 22, 2018, 1:37 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69430/
> ---
> 
> (Updated Nov. 22, 2018, 1:37 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3409
> https://issues.apache.org/jira/browse/SQOOP-3409
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. This was 
> caused as the target directory cleanup and name reset happened before the 
> temprootdir cleanup however the temprootdir name depends on the target dir 
> name in the tests.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 2fc606115196a7a2b6088be104e2a421888f8798 
> 
> 
> Diff: https://reviews.apache.org/r/69430/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew s3Test -Ds3.bucket.url= 
> -Ds3.generator.command=, all the used 
> temprootdirs have been cleaned up
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69429: Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69429/#review210799
---


Ship it!




Ship It!

- Fero Szabo


On Nov. 22, 2018, 1:11 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69429/
> ---
> 
> (Updated Nov. 22, 2018, 1:11 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3408
> https://issues.apache.org/jira/browse/SQOOP-3408
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Documented forkEvery.default in COMPILING.txt.
> Needed to move the definition of the kerberizedTest task below 
> tasks.withType(Test) block to preserve forkEvery 1 setting.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 0383707f689102a3a543d94646cfaaf21710 
>   build.gradle 954935daeaaaf45e1b2fd83f74e11f5ed2d58377 
> 
> 
> Diff: https://reviews.apache.org/r/69429/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew test : runs the test task with forkEvery=0
> ./gradlew -DforkEvery.default=5 test : runs the test taks with forkEvery=5
> 
> ./gradlew kerberizedTest : runs the kerberizedTest task with forkEvery=1
> ./gradlew -DforkEvery.default=5 kerberizedTest : runs the kerberizedTest task 
> with forkEvery=1, so the forkEvery.default parameter does not affect 
> kerberizedTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

> On Nov. 22, 2018, 8:29 a.m., Boglarka Egyed wrote:
> > src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
> > Lines 24 (patched)
> > <https://reviews.apache.org/r/69407/diff/6/?file=2109532#file2109532line24>
> >
> > Renamed files are shown as new files now which compromises the diff. 
> > Could you please take a look and regenerate the diff?
> 
> Boglarka Egyed wrote:
> Sorry, I wanted to say that it corrupts the diff.

Fixed with black-belt git magic, with the help of Szabi. ;)


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210785
---


On Nov. 22, 2018, 1:39 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 22, 2018, 1:39 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe1 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebeb 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162d 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd014688 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/7/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 22, 2018, 1:39 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7defe1 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdebeb 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
22567162d 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
ebd014688 


Diff: https://reviews.apache.org/r/69407/diff/7/

Changes: https://reviews.apache.org/r/69407/diff/6-7/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

> On Nov. 21, 2018, 2:05 p.m., Boglarka Egyed wrote:
> > src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/69407/diff/5/?file=2109510#file2109510line1>
> >
> > Apache headers are missing from new files.

Thanks for pointing this out, should be ok now. I didn't make any other code 
change, just added the licence headers, but Reviewboard somehow shows more...


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210762
---


On Nov. 21, 2018, 3 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 21, 2018, 3 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7def 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdeb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/SqlServerImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesImportTestBase.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SplitByImportTestBase.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 2256716 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd0146 
>   src/test/org/apache/sqoop/testutil/adapter/MysqlDatabaseAdapter.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/SqlServerDatabaseAdapter.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/6/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 21, 2018, 3 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cb 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbb 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7def 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdeb 
  
src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/SqlServerImportJobTestConfiguration.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesImportTestBase.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SplitByImportTestBase.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
2256716 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java ebd0146 
  src/test/org/apache/sqoop/testutil/adapter/MysqlDatabaseAdapter.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/SqlServerDatabaseAdapter.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69407/diff/6/

Changes: https://reviews.apache.org/r/69407/diff/5-6/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Re: Review Request 69414: Sqoop should not try to execute test category interfaces as tests with Ant

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69414/#review210763
---


Ship it!




Ship It!

- Fero Szabo


On Nov. 21, 2018, 12:22 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69414/
> ---
> 
> (Updated Nov. 21, 2018, 12:22 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3406
> https://issues.apache.org/jira/browse/SQOOP-3406
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> When Ant third party test suite is being run Ant tries to execute the test 
> category interfaces too because they end with the 'Test' postfix.
> 
> These "tests" obviously fail so we need to make sure that Ant does not 
> execute them.
> 
> 
> Diffs
> -
> 
>   build.xml 995a513040f85b6c2043a977a09e93b56913bbed 
> 
> 
> Diff: https://reviews.apache.org/r/69414/diff/2/
> 
> 
> Testing
> ---
> 
> ant unit and third party test
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

[jira] [Comment Edited] (SQOOP-3407) Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694648#comment-16694648
 ] 

Fero Szabo edited comment on SQOOP-3407 at 11/21/18 12:22 PM:
--

Hi [~vasas],

I've committed your patch, thank you for your contribution!

You can close the related review request.


was (Author: fero):
Hi [~vasas],

Thank you for your contribution!

You can close the related review request.

> Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static 
> fields
> 
>
> Key: SQOOP-3407
> URL: https://issues.apache.org/jira/browse/SQOOP-3407
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3407.patch
>
>
> BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are 
> public static fields which get initialized once at the JVM startup and store 
> the paths for the test temp and warehouse directories.
> The problem is that HBase test cases change the value of the test.build.data 
> system property which can cause tests using these static fields to fail.
> Since we do not own the code in HBase which changes the system property we 
> need to turn these static fields into methods which evaluate the 
> test.build.data system property every time they invoked which will make sure 
> that the invoking tests will be successful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69413: Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69413/#review210754
---


Ship it!




Lgtm.

- Fero Szabo


On Nov. 20, 2018, 5:29 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69413/
> ---
> 
> (Updated Nov. 20, 2018, 5:29 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3407
> https://issues.apache.org/jira/browse/SQOOP-3407
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are 
> public static fields which get initialized once at the JVM startup and store 
> the paths for the test temp and warehouse directories.
> 
> The problem is that HBase test cases change the value of the test.build.data 
> system property which can cause tests using these static fields to fail.
> 
> Since we do not own the code in HBase which changes the system property we 
> need to turn these static fields into methods which evaluate the 
> test.build.data system property every time they invoked which will make sure 
> that the invoking tests will be successful.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/TestIncrementalImport.java 
> dbdd05c13e77af514bd996a92f7ebea3a27aedd5 
>   src/test/org/apache/sqoop/TestMerge.java 
> b283174b8b3df7c16c496795fcbae2f91dd1c375 
>   src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java 
> 9c1e9f9a93323655bc313303bf84d566b551ee00 
>   src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java 
> df1840b37ce29ffb303b31e1fcbfe4c5842e7c36 
>   src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java 
> 71d6971489e489ae501739fdad5a7409375b6ec1 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
> ea7942f62d623895f242e69e77cf9920bbb7e18c 
>   src/test/org/apache/sqoop/orm/TestClassWriter.java 
> 59a8908f13c51b9caca42e8602413ee0b8634b0a 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
> e23aad3ee997780e5708e9180550339d834b74d9 
> 
> 
> Diff: https://reviews.apache.org/r/69413/diff/1/
> 
> 
> Testing
> ---
> 
> Executed unit and third party tests.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69414: Sqoop should not try to execute test category interfaces as tests with Ant

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69414/#review210753
---




build.xml
Lines 950 (patched)
<https://reviews.apache.org/r/69414/#comment295509>

CubridTest.java is excluded twice?


- Fero Szabo


On Nov. 20, 2018, 5:32 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69414/
> ---
> 
> (Updated Nov. 20, 2018, 5:32 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3406
> https://issues.apache.org/jira/browse/SQOOP-3406
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> When Ant third party test suite is being run Ant tries to execute the test 
> category interfaces too because they end with the 'Test' postfix.
> 
> These "tests" obviously fail so we need to make sure that Ant does not 
> execute them.
> 
> 
> Diffs
> -
> 
>   build.xml 995a513040f85b6c2043a977a09e93b56913bbed 
> 
> 
> Diff: https://reviews.apache.org/r/69414/diff/1/
> 
> 
> Testing
> ---
> 
> ant unit and third party test
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 21, 2018, 10:20 a.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7defe 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdebe 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
22567162 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java ebd01468 


Diff: https://reviews.apache.org/r/69407/diff/5/

Changes: https://reviews.apache.org/r/69407/diff/4-5/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 21, 2018, 9:31 a.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7defe1 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdebeb 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
22567162d 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
ebd014688 


Diff: https://reviews.apache.org/r/69407/diff/4/

Changes: https://reviews.apache.org/r/69407/diff/3-4/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 21, 2018, 9:29 a.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/java/org/apache/sqoop/hive/HiveTypes.java 554a03608 
  src/java/org/apache/sqoop/hive/TableDefWriter.java b21dfe534 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java 8cfd776d5 


Diff: https://reviews.apache.org/r/69407/diff/3/

Changes: https://reviews.apache.org/r/69407/diff/2-3/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 21, 2018, 9:07 a.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
---

Refactored test class names to the same as used in the categorization


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/java/org/apache/sqoop/hive/HiveTypes.java 554a03608 
  src/java/org/apache/sqoop/hive/TableDefWriter.java b21dfe534 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java 8cfd776d5 


Diff: https://reviews.apache.org/r/69407/diff/2/

Changes: https://reviews.apache.org/r/69407/diff/1-2/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7defe1 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdebeb 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
22567162d 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
ebd014688 


Diff: https://reviews.apache.org/r/69407/diff/1/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

[jira] [Updated] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3405:
--
Description: 
Follow the example of the abstract class SavedJobsTestBase and it's subclasses!

We need this to be able to add test categories (so for Travis integration) as 
well.

  was:Follow the example of the abstract class SavedJobsTestBase and it's 
subclasses!


> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!
> We need this to be able to add test categories (so for Travis integration) as 
> well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3405:
--
Summary: Refactor: break up Parameterized tests on a per database basis  
(was: Refactor: break up NumericTypesImportTest to be executable on a per 
database basis)

> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3405) Refactor: break up NumericTypesImportTest to be executable on a per database basis

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3405:
-

 Summary: Refactor: break up NumericTypesImportTest to be 
executable on a per database basis
 Key: SQOOP-3405
 URL: https://issues.apache.org/jira/browse/SQOOP-3405
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Fero Szabo
Assignee: Fero Szabo


Follow the example of the abstract class SavedJobsTestBase and it's subclasses!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69346: Categorize all tests in the project

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69346/#review210576
---


Ship it!




Hi Szabi,

lgtm, just a minor suggestion.


COMPILING.txt
Line 465 (original), 633 (patched)
<https://reviews.apache.org/r/69346/#comment295281>

I believe you require a fully qualified classname here, right? A concrete 
example here might come in handy. What do you think?


- Fero Szabo


On Nov. 15, 2018, 5:45 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69346/
> ---
> 
> (Updated Nov. 15, 2018, 5:45 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3404
> https://issues.apache.org/jira/browse/SQOOP-3404
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> - All tests are categorized now
> - Introduced S3Test category as a subcategory of ThirdPartyTest
> - Reorganized test tasks: we have unitTest, integrationTest, kerberizedTest, 
> test, s3Test, allTest tasks now
> - jacocoTestReport task is fixed to contain the coverage information of the 
> kerberizedTest task too. This is needed because the kerberizedTest needs the 
> forkEvery parameter to be set to 1 and because of that it has to be a 
> separate task which generates separate coverage information too. However it 
> is automatically triggered after the test task so the invocation is more 
> convenient for the tester.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 835ba33b1e89158bed0e05698b188ab3323eb881 
>   build.gradle cb9eeca74bbf278c3e5fd15de608d8c37c917ddb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 
> c6fe4f2e8a80c96ad667f4fe4a26510af96562dc 
>   src/test/org/apache/sqoop/manager/TestMainframeManager.java 
> c84f05f660c396a06a5031e00abdae77ffbcf2aa 
>   
> src/test/org/apache/sqoop/manager/oracle/TestOraOopDBInputSplitGetDebugDetails.java
>  6f33ad3b650436b7f268b4ef5bfd451bd5e6958e 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableExportMapper.java
>  5e558717c0d43301ecbf81a37d5ee3fd35756d65 
>   
> src/test/org/apache/sqoop/mapreduce/db/netezza/TestNetezzaExternalTableImportMapper.java
>  1a6943786834d27f27523f484d76cf678f18cf48 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java
>  b4cba28c3611400b5c4227a5166b6c91e9152dc4 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
>  521a04266e8806321fe7aa6a89c064f369174523 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> 7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 
>   src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
> 0c3161e5a783446e35f4754124f86715d103ec0b 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> 3a0d6365dc20f8eef5bdd67a4a2dc9c68ff74d7f 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> 5faf59ea80c48fe025294cabd100e7d176032138 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> a4f986423ea299716a29f9d02f7c8453a7f2ba02 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> d271588c5af060bbc3d301a845f45c46d0f6a2ba 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> 52d89c775b5f1219471df44d222fd92a59ed408c 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> 39238c5fab56b54a85dde5aed0d4bb2c77382fa6 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> 597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java 
> c9785d816d4a7a5870d74c51a9faa229f6d3818e 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> bba8b74ebe639df26e977abf377f4904144dcfaa 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 114f97cbb8857a7633cae5d030769ac4a90e36aa 
>   src/test/org/apache/sqoop/testcategories/thirdpartytest/S3Test.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
> 7745f1b07e6d6c457b0164deeace12587ec058d0 
> 
> 
> Diff: https://reviews.apache.org/r/69346/diff/3/
> 
> 
> Testing
> ---
> 
> ./gradlew unitTest
> ./gradlew integrationTest
> ./gradlew kerberizedTest
> ./gradlew ... s3Test
> ./gradlew test
> ./gradlew ... thirdPartyTest
> ./gradlew allTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
---

(Updated Nov. 12, 2018, 4:33 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3382
https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description
---

This patch is about adding support for fixed point decimal types in parquet 
import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 
in SQOOP-3381: we just need to register the GenericDataSupplier with 
AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
under the hood to write parquet.

I also moved around and renamed the classes involved in this change so their 
name and package reflect their purpose.

** Note: A key design decision can be seen in the ImportJobTestConfiguration 
interface **
- I decided to create a new function to get the expected results for each file 
format, since we seldom add new fileformats. 
- However this also enforces future configurations to always define their 
expected result for every file forma or throw a NotImplementedException should 
they lack the support for one.
- The alternative for this is to define the fileLayout as an input parameter 
instead. This would allow for better extendability.
_Please share your thoughts on this!_


Diffs (updated)
-

  src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 e82154309 
  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b9 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3bc 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967f 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912a 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c0 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c171 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41bd 
  src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 f137b56b7 
  
src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java 
PRE-CREATION 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 


Diff: https://reviews.apache.org/r/69060/diff/4/

Changes: https://reviews.apache.org/r/69060/diff/3-4/


Testing
---

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

> On Nov. 9, 2018, 2:26 p.m., Boglarka Egyed wrote:
> > src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java
> > Lines 220 (patched)
> > <https://reviews.apache.org/r/69060/diff/3/?file=2106486#file2106486line224>
> >
> > I think these tests could be parameterized as they are doing the same 
> > but with different file formats (Avro and Parquet).

Hi Bogi,

Thanks for the review!

There is a tiny difference: to enable logical types in parquet, there is new 
flag (sqoop.parquet.logical_types.decimal.enable), i.e. only used in the 
parquet tests. 

I'd keep this code as is, as deduplication might lead to spaghetti code here 
(since these are different features after all).

Even though this is a bit of a compromise, I'd like to drop this issue if 
that's OK with you (?)


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review210439
---


On Nov. 8, 2018, 3:34 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Nov. 8, 2018, 3:34 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c069888 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b9 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967f 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912a 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c0 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c171 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41bd 
>   
> src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
>  f137b56b7 
>   
> src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/3/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

[jira] [Resolved] (SQOOP-3403) Sqoop2: Add Fero Szabo to committer list in our pom file

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo resolved SQOOP-3403.
---
Resolution: Fixed

> Sqoop2: Add Fero Szabo to committer list in our pom file
> 
>
> Key: SQOOP-3403
> URL: https://issues.apache.org/jira/browse/SQOOP-3403
> Project: Sqoop
>  Issue Type: Task
>Affects Versions: 1.99.8
>Reporter: Boglarka Egyed
>    Assignee: Fero Szabo
>Priority: Major
>
> Now that [~fero] is committer we should update our committer list in the root 
> pom.xml file:



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
---

(Updated Nov. 8, 2018, 3:34 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
---

The most notable change in this newest changesetis around verification: 
- as we discussed with Szabi: since decimal conversion is put in place it's not 
enough to assert on the data itself (as the data appears as String in-memory, 
it could be String on the disk as well)
- so, I've added a new function that checks the schema for decimal types as well


Bugs: SQOOP-3382
https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description
---

This patch is about adding support for fixed point decimal types in parquet 
import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 
in SQOOP-3381: we just need to register the GenericDataSupplier with 
AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
under the hood to write parquet.

I also moved around and renamed the classes involved in this change so their 
name and package reflect their purpose.

** Note: A key design decision can be seen in the ImportJobTestConfiguration 
interface **
- I decided to create a new function to get the expected results for each file 
format, since we seldom add new fileformats. 
- However this also enforces future configurations to always define their 
expected result for every file forma or throw a NotImplementedException should 
they lack the support for one.
- The alternative for this is to define the fileLayout as an input parameter 
instead. This would allow for better extendability.
_Please share your thoughts on this!_


Diffs (updated)
-

  src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c069888 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b9 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3bc 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967f 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912a 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c0 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c171 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41bd 
  src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 f137b56b7 
  
src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java 
PRE-CREATION 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 


Diff: https://reviews.apache.org/r/69060/diff/3/

Changes: https://reviews.apache.org/r/69060/diff/2-3/


Testing
---

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

> On Oct. 26, 2018, 7:40 a.m., Szabolcs Vasas wrote:
> > src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java
> > Lines 292 (patched)
> > <https://reviews.apache.org/r/69060/diff/2/?file=2099873#file2099873line296>
> >
> > Since the readAll* methods of ParquetReader close the reader this 
> > method could be simplified to something like this:
> > 
> > private void verifyParquetFile() {
> > ParquetReader reader = new ParquetReader(new Path(getWarehouseDir() 
> > + "/" + getTableName()), getConf());
> > assertEquals(asList(configuration.getExpectedResultsForParquet()), 
> > reader.readAllInCsv());
> >   }

I think this no longer applies since I access the convertToCsv method 4 lines 
later of the same reader.
That method could be made public static as well, though I'm unsure of such a 
utility method would be considered best practice. (?)


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review210083
---


On Nov. 8, 2018, 3:34 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Nov. 8, 2018, 3:34 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c069888 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b9 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967f 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912a 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c0 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c171 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41bd 
>   
> src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
>  f137b56b7 
>   
> src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/3/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: [ANNOUNCE] New Sqoop PMC member - Szabolcs Vasas

[Fero Szabo]

Hi Szabi,

Congratulations!

It's good to see your hard work came to fruition!

I am looking forward to (continue) working with you!

Best Regards,
Fero


On Tue, Nov 6, 2018 at 10:49 AM Boglarka Egyed  wrote:

> On behalf of the Apache Sqoop PMC, I am pleased to welcome Szabolcs Vasas
> as
> a new Sqoop PMC Member. Please join me in congratulating him!
>
> Szabolcs has countless code contributions[1] as well as provides thorough
> reviews
> for others constantly[2]. He continuously offers help to new contributors
> enabling
> the project to grow and he also demonstrated a huge interest to shape the
> project
> with helping and scoping previous and upcoming releases.
>
> Szabolcs's hard work is much appreciated and we look forward to his
> continued contributions!
>
> 1: https://s.apache.org/nzgU
> 2: https://reviews.apache.org/users/vasas/reviews/
>
> Kind Regards,
> Bogi
>


-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

Re: Review Request 69199: Create tests for SQOOP-2949, quote escaping in split-by

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69199/
---

(Updated Oct. 31, 2018, 1:51 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3400
https://issues.apache.org/jira/browse/SQOOP-3400


Repository: sqoop-trunk


Description
---

Integration tests for SQOOP-2949.


Diffs (updated)
-

  src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java 22bbfe68 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69199/diff/6/

Changes: https://reviews.apache.org/r/69199/diff/5-6/


Testing
---

This is the testing part for a fix that lacked testing. 
gradle test and gradle 3rdpartytests.


Thanks,

Fero Szabo

Re: Review Request 69199: Create tests for SQOOP-2949, quote escaping in split-by

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69199/
---

(Updated Oct. 31, 2018, 12:54 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
---

Implemented fixes.


Bugs: SQOOP-3400
https://issues.apache.org/jira/browse/SQOOP-3400


Repository: sqoop-trunk


Description
---

Integration tests fro SQOOP-2949.


Diffs (updated)
-

  src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java 22bbfe68 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/69199/diff/4/

Changes: https://reviews.apache.org/r/69199/diff/3-4/


Testing
---

This is the testing part for a fix that lacked testing. 
gradle test and gradle 3rdpartytests.


Thanks,

Fero Szabo

Re: Review Request 69199: Create tests for SQOOP-2949, quote escaping in split-by

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69199/
---

(Updated Oct. 30, 2018, 4:26 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
---

As we discussed offline with Szabolcs, I've updated this diff to include the 
fix for SQOOP-2949.

Also, simplified tests a bit by asserting for Parquet output.


Bugs: SQOOP-3400
https://issues.apache.org/jira/browse/SQOOP-3400


Repository: sqoop-trunk


Description
---

Integration tests fro SQOOP-2949.


Diffs (updated)
-

  src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java 22bbfe68 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 PRE-CREATION 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java fe6ba831 


Diff: https://reviews.apache.org/r/69199/diff/2/

Changes: https://reviews.apache.org/r/69199/diff/1-2/


Testing
---

This is the testing part for a fix that lacked testing. 
gradle test and gradle 3rdpartytests.


Thanks,

Fero Szabo

[jira] [Commented] (SQOOP-2949) SQL Syntax error when split-by column is of character type and min or max value has single quote inside it

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667424#comment-16667424
 ] 

Fero Szabo commented on SQOOP-2949:
---

Hi [~gireeshp],

As we agreed offline, I've developed tests for this fix in SQOOP-3400. I've 
also posted your change on review board, because it's required for the tests. 
Hope you don't mind! (I mentioned that you developed it, wouldn't want to steal 
the credit!)

In any case, please feel free to review the tests if you can find the time!

Bests,

Fero

> SQL Syntax error when split-by column is of character type and min or max 
> value has single quote inside it
> --
>
> Key: SQOOP-2949
> URL: https://issues.apache.org/jira/browse/SQOOP-2949
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Sqoop 1.4.6
> Run on Hadoop 2.6.0
> On Ubuntu
>Reporter: Gireesh Puthumana
>Assignee: Gireesh Puthumana
>Priority: Major
>
> Did a sqoop import from mysql table "emp", with split-by column "ename", 
> which is a varchar(100) type.
> +Used below command:+
> sqoop import --connect jdbc:mysql://localhost/testdb --username root 
> --password * --table emp --m 2 --target-dir /sqoopTest/5 --split-by ename;
> +Ename has following records:+
> | ename   |
> | gireesh |
> | aavesh  |
> | shiva'  |
> | jamir   |
> | balu|
> | santosh |
> | sameer  |
> Min value is "aavesh" and max value is "shiva'" (please note the single quote 
> inside max value).
> When run, it tried to execute below query in mapper 2 and failed:
> SELECT `ename`, `eid`, `deptid` FROM `emp` AS `emp` WHERE ( `ename` >= 
> 'jd聯聭聪G耀' ) AND ( `ename` <= 'shiva'' )
> +Stack trace:+
> {quote}
> 2016-06-05 16:54:06,749 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error 
> in your SQL syntax; check the manual that corresponds to your MySQL server 
> version for the right syntax to use near ''shiva'' )' at line 1
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>   at com.mysql.jdbc.Util.getInstance(Util.java:387)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:942)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>   at 
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69199: Create tests for SQOOP-2949, quote escaping in split-by

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69199/
---

Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3400
https://issues.apache.org/jira/browse/SQOOP-3400


Repository: sqoop-trunk


Description
---

Integration tests fro SQOOP-2949.


Diffs
-

  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c17 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41b 
  
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
 PRE-CREATION 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java fe6ba831 


Diff: https://reviews.apache.org/r/69199/diff/1/


Testing
---

This is the testing part for a fix that lacked testing. 
gradle test and gradle 3rdpartytests.


Thanks,

Fero Szabo

Review Request 69200: SQL Syntax error when split-by column is of character type and min or max value has single quote inside it

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69200/
---

Review request for Sqoop, Boglarka Egyed, Fero Szabo, and Szabolcs Vasas.


Bugs: SQOOP-2949
https://issues.apache.org/jira/browse/SQOOP-2949


Repository: sqoop-trunk


Description
---

This fix was developped by Gireesh Puthumana.

This is a fix for text splitting. 
- Previously, if a split boundary record had a quote in inside, an error 
occured because of the lack of escaping. 
- This change adds the escaping for quotes via standard SQL notation that 
should work across all databases.


Diffs
-

  src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java 22bbfe68 


Diff: https://reviews.apache.org/r/69200/diff/1/


Testing
---

New integration tests were developed in SQOOP-3400.


Thanks,

Fero Szabo

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

> On Oct. 24, 2018, 2:31 p.m., Szabolcs Vasas wrote:
> > Hi Feró,
> > 
> > Thank you for submitting this improvement!
> > I have left some comments, see them below.
> > Apart from that I think we need to test explicitly that if the 
> > sqoop.parquet.logical_types.decimal.enable flag is true then the Parquet 
> > file contains a decimal value and otherwise it contains a string value.
> > 
> > NumericTypesImportTest asserts on string values so it is not able to verify 
> > this, most of the tests passed even if I commented out the content of the 
> > addEnableParquetDecimal method.

I'll look into this one.

I'm thinking that using the org.apache.sqoop.util.ParquetReader#readAll method 
could help (since it returns GenericRecords), though I'm not sure. I'll somehow 
need to actually turn off the conversion and check for the bytes. Any 
suggestions?


> On Oct. 24, 2018, 2:31 p.m., Szabolcs Vasas wrote:
> > src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
> > Lines 115-119 (patched)
> > <https://reviews.apache.org/r/69060/diff/2/?file=2099869#file2099869line115>
> >
> > Is it possible to move this to 
> > org.apache.sqoop.mapreduce.parquet.hadoop.HadoopParquetImportJobConfigurator#configureMapper?
> > That would be consistent with the way we configure the Parquet imports 
> > but I am not sure the effect would remain the same.

Yes, makes sense.


> On Oct. 24, 2018, 2:31 p.m., Szabolcs Vasas wrote:
> > src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
> > Lines 56 (patched)
> > <https://reviews.apache.org/r/69060/diff/2/?file=2099870#file2099870line56>
> >
> > Are we sure that adding the logical type conversion only here is enough?
> > In case of Avro it is also added in 
> > org.apache.sqoop.mapreduce.AvroOutputFormat#getRecordWriter which gets 
> > invoked in every mapper so I assume that we have to add the conversion in 
> > every mapper in case of Parquet files too.

My understanding is that this method is invoked in every mapper. (it's doc 
suggest this as well: "Called once at the beginning of the task.")
Where else would put this statement?


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review209954
---


On Oct. 24, 2018, 12:25 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Oct. 24, 2018, 12:25 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967 
>   

Re: Review Request 69139: TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69139/#review209969
---


Ship it!




Lgtm. I've only run this one test though.

- Fero Szabo


On Oct. 24, 2018, 10:58 a.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69139/
> ---
> 
> (Updated Oct. 24, 2018, 10:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3399
> https://issues.apache.org/jira/browse/SQOOP-3399
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw 
> NullPointerException when the credential generator command is not provided 
> since it fails the test with Gradle.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> e03eb64ef 
> 
> 
> Diff: https://reviews.apache.org/r/69139/diff/1/
> 
> 
> Testing
> ---
> 
> Executed the test with both and and gradle, with and without S3 credential 
> generator provided.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69066: SQOOP-3390: Document S3Guard usage with Sqoop

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69066/#review209964
---


Ship it!




Bogi, thanks for enhancing our documentation as well!

Both ant and gw docs are successful on my side, and the output look good.

- Fero Szabo


On Oct. 17, 2018, 4:49 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69066/
> ---
> 
> (Updated Oct. 17, 2018, 4:49 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3390
> https://issues.apache.org/jira/browse/SQOOP-3390
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Document Hadoop's S3Guard usage with Sqoop to overcome Amazon S3's eventual 
> consistency.
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt c54b26bc5ef71f8cd7d18ce6eb98a296dbffed92 
> 
> 
> Diff: https://reviews.apache.org/r/69066/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69141: Tests using HiveMiniCluster can be unstable on some platforms

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69141/#review209963
---


Ship it!




gradlew test and thirdpartytest are successful on my computer.

- Fero Szabo


On Oct. 24, 2018, 12:25 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69141/
> ---
> 
> (Updated Oct. 24, 2018, 12:25 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3398
> https://issues.apache.org/jira/browse/SQOOP-3398
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Since the last Hive upgrade TestHiveMiniCluster fails on some platforms 
> because and older version of the ASM library is picked up.
> 
> The task is to exclude the older ASM library in ivy and gradle to make sure 
> the test passes on all platforms.
> 
> 
> Diffs
> -
> 
>   build.gradle 2340bce7519a46b203a287a4b5160c62e0c09509 
>   ivy.xml 6805fc329d44bcc0707e7cab67f3749a42e6f769 
> 
> 
> Diff: https://reviews.apache.org/r/69141/diff/1/
> 
> 
> Testing
> ---
> 
> Executed unit and third party tests with both ant and gradle.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 68064: SQOOP-3355 Document SQOOP-1905 DB2 --schema option

[Fero Szabo via Review Board]

> On Oct. 24, 2018, 12:21 p.m., Szabolcs Vasas wrote:
> > src/docs/user/connectors.txt
> > Lines 41 (patched)
> > <https://reviews.apache.org/r/68064/diff/1/?file=2064419#file2064419line41>
> >
> > Are you sure import-all-tables supports --schema properly?
> > By looking at the code of this command it seems to me that Sqoop uses 
> > the list-tables command to get the table names it wants to import but then 
> > it invokes the import command in a loop to import all the tables. So I 
> > think that even if the list-tables shows a table the import command will 
> > not be able to pick it up.

Hi Szabi,

Yes, I've tested this in July, when I created the review. Now this reminds me 
to write down everything. :)

Anyway, I've tested again, and this works as expected. What I did was: fire up 
a cluster, created a couple of tables in a DB2 instance and ran the command. 2 
different schemas, all the data form all of the tables are in hdfs. I'd say 
this is proper behavior.

Now I'm not sure why/how this feature wasn't implemented for a (simple) import, 
but it just wasn't. The error message suggests that sqoop tries to use the 
default schema, i.e. DB2INST1. We can open a Jira if there isn't one.

I'm dropping this issue.


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68064/#review209952
-------


On July 26, 2018, 2:58 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68064/
> ---
> 
> (Updated July 26, 2018, 2:58 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3355
> https://issues.apache.org/jira/browse/SQOOP-3355
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Adding documentation for a previously implemented feature. This one is quite 
> simple.
> 
> 
> Diffs
> -
> 
>   src/docs/user/connectors.txt 59e3e00b 
> 
> 
> Diff: https://reviews.apache.org/r/68064/diff/1/
> 
> 
> Testing
> ---
> 
> ant docs, 
> + unit and 3rd party tests, though these shouldn't be affected.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
---

(Updated Oct. 24, 2018, 12:25 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3382
https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description (updated)
---

This patch is about adding support for fixed point decimal types in parquet 
import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 
in SQOOP-3381: we just need to register the GenericDataSupplier with 
AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
under the hood to write parquet.

I also moved around and renamed the classes involved in this change so their 
name and package reflect their purpose.

** Note: A key design decision can be seen in the ImportJobTestConfiguration 
interface **
- I decided to create a new function to get the expected results for each file 
format, since we seldom add new fileformats. 
- However this also enforces future configurations to always define their 
expected result for every file forma or throw a NotImplementedException should 
they lack the support for one.
- The alternative for this is to define the fileLayout as an input parameter 
instead. This would allow for better extendability.
_Please share your thoughts on this!_


Diffs
-

  src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c17 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41b 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 


Diff: https://reviews.apache.org/r/69060/diff/2/


Testing
---

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo

[jira] [Created] (SQOOP-3400) Create tests for SQOOP-2949, quote escaping in split-by

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3400:
-

 Summary: Create tests for SQOOP-2949, quote escaping in split-by
 Key: SQOOP-3400
 URL: https://issues.apache.org/jira/browse/SQOOP-3400
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Fero Szabo
Assignee: Fero Szabo






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69070: SQOOP-3394: External Hive table tests should use unique external dir names

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69070/#review209779
---


Ship it!




Lgtm

- Fero Szabo


On Oct. 18, 2018, 5:49 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69070/
> ---
> 
> (Updated Oct. 18, 2018, 5:49 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3394
> https://issues.apache.org/jira/browse/SQOOP-3394
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Current external Hive table tests on S3 uses the same external directory name 
> in every unit test cases which can cause problems during running them in an 
> automated environment. These names should be unique in every test cases.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 97d53bbaa7c72d2ad1b890d7a8367c45a3e2b95c 
> 
> 
> Diff: https://reviews.apache.org/r/69070/diff/2/
> 
> 
> Testing
> ---
> 
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Created] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3396:
-

 Summary: Add parquet numeric support for Parquet in Hive import
 Key: SQOOP-3396
 URL: https://issues.apache.org/jira/browse/SQOOP-3396
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Fero Szabo






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo reassigned SQOOP-3396:
-

Assignee: Fero Szabo

> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
>  Issue Type: Sub-task
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3382) Add parquet numeric support for Parquet in hdfs import

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3382:
--
Summary: Add parquet numeric support for Parquet in hdfs import  (was: Add 
parquet numeric support and reuse existing Avro numeric tests Parquet)

> Add parquet numeric support for Parquet in hdfs import
> --
>
> Key: SQOOP-3382
> URL: https://issues.apache.org/jira/browse/SQOOP-3382
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> The current Avro numeric tests are suitable to be used as Parquet tests, with 
> very minor modifications, as parquet can be written with the same input and 
> nearly the same args. Since we are writing Parquet with it's Avro support, it 
> would be good to cover this code with the same, or similar tests (including 
> the edge cases related to padding, missing scale and precision cases).
> Differences are:
>  * the expected output, since stored in a parquet file is different.
>  * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
---

(Updated Oct. 18, 2018, 5:40 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
---

Added a new flag for enabling logical types with parquet. This should allow 
backward compatibility.


Summary (updated)
-

SQOOP-3382 Add parquet numeric support for Parquet in hdfs import


Bugs: SQOOP-3382
https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description
---

This patch is about adding support for fixed point decimal types in parquet 
import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 
in SQOOP-3381: we just need to register the GenericDataSupplier with 
AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
under the hood to write parquet.

I also moved around and renamed the classes involved in this change so their 
name and package reflect their purpose.


Diffs (updated)
-

  src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c17 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41b 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 


Diff: https://reviews.apache.org/r/69060/diff/2/

Changes: https://reviews.apache.org/r/69060/diff/1-2/


Testing
---

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo

Re: Review Request 69063: SQOOP-3391: Test storing AWS credentials in Hadoop CredentialProvider during import

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69063/#review209740
---


Ship it!




lgtm

- Fero Szabo


On Oct. 18, 2018, 11:45 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69063/
> ---
> 
> (Updated Oct. 18, 2018, 11:45 a.m.)
> 
> 
> Review request for Sqoop, Fero Szabo, Ferenc Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3391
> https://issues.apache.org/jira/browse/SQOOP-3391
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Test storing AWS credentials in Hadoop CredentialProvider during import in 
> case of
> - CredntialProvider with default password
> - CredntialProvider with password stored in environment variable
> - CredntialProvider with password file
> Added test cases for happy and sad paths as well.
> 
> Added a new test dependency both in Ant and Gradle for setting environment 
> variables in tests easily.
> 
> 
> Diffs
> -
> 
>   build.gradle 7a0712e3242e31ef2593c34f469f9136cf5dc85d 
>   build.xml f3975317140e66c700d85231669ccb2b70367f80 
>   conf/password-file.txt PRE-CREATION 
>   conf/wrong-password-file.txt PRE-CREATION 
>   gradle.properties 4808ec7d090b9732f9246f21e44bd736adf6efd0 
>   ivy.xml 91157ca74bee3b50269564ddb747638946e45a7e 
>   ivy/libraries.properties 2ca95ee99c09fe1aaff6797a6ee0958ac1977663 
>   src/java/org/apache/sqoop/util/password/CredentialProviderHelper.java 
> 1d6481a0697db2fc0ffeb1b012bb143beb615bc0 
>   src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> c9d17bc728d6a229e32c157b56268d6418b3de94 
> 
> 
> Diff: https://reviews.apache.org/r/69063/diff/2/
> 
> 
> Testing
> ---
> 
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> ant clean test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Review Request 69060: SQOOP-3382 Add parquet numeric support and reuse existing Avro numeric tests Parquet

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
---

Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3382
https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description
---

This patch is about adding support for fixed point decimal types in parquet 
import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 
in SQOOP-3381: we just need to register the GenericDataSupplier with 
AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
under the hood to write parquet.

I also moved around and renamed the classes involved in this change so their 
name and package reflect their purpose.


Diffs
-

  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
ff13dc3b 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
 182d2967 
  
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
 e9bf9912 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
 b7bad08c 
  
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
 465e61f4 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
 66715c17 
  
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
 ec4db41b 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 


Diff: https://reviews.apache.org/r/69060/diff/1/


Testing
---

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo

[jira] [Updated] (SQOOP-3382) Add parquet numeric support and reuse existing Avro numeric tests Parquet

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3382:
--
Summary: Add parquet numeric support and reuse existing Avro numeric tests 
Parquet  (was: Add parquet numeric support and refactor existing Avro numeric 
tests for reusability (with Parquet))

> Add parquet numeric support and reuse existing Avro numeric tests Parquet
> -
>
> Key: SQOOP-3382
> URL: https://issues.apache.org/jira/browse/SQOOP-3382
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> The current Avro numeric tests are suitable to be used as Parquet tests, with 
> very minor modifications, as parquet can be written with the same input and 
> nearly the same args. Since we are writing Parquet with it's Avro support, it 
> would be good to cover this code with the same, or similar tests (including 
> the edge cases related to padding, missing scale and precision cases).
> Differences are:
>  * the expected output, since stored in a parquet file is different.
>  * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3382) Add parquet numeric support and refactor existing Avro numeric tests for reusability (with Parquet)

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3382:
--
Summary: Add parquet numeric support and refactor existing Avro numeric 
tests for reusability (with Parquet)  (was: Reafactor existing Avro numeric 
tests for reusability (with Parquet))

> Add parquet numeric support and refactor existing Avro numeric tests for 
> reusability (with Parquet)
> ---
>
> Key: SQOOP-3382
> URL: https://issues.apache.org/jira/browse/SQOOP-3382
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> The current Avro numeric tests are suitable to be used as Parquet tests, with 
> very minor modifications, as parquet can be written with the same input and 
> nearly the same args. Since we are writing Parquet with it's Avro support, it 
> would be good to cover this code with the same, or similar tests (including 
> the edge cases related to padding, missing scale and precision cases).
> Differences are:
>  * the expected output, since stored in a parquet file is different.
>  * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Sqoop PMC member - Boglarka Egyed

[Fero Szabo]

Hi Bogi,

Well earned, congratulations!

Live long and keep contributing! :)

Cheers,
Fero


On Tue, Oct 16, 2018 at 10:28 AM Jarek Jarcec Cecho 
wrote:

> On behalf of the Apache Sqoop PMC, I am excited to welcome Boglarka Egyed
> as new Sqoop PMC Member. Please join me in congratulating her!
>
> Jarcec
>
>

-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Oct. 16, 2018, 9:37 a.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version and a whole lot 
of libraries to newer versions with it.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

A few things to note:
- hadoop's version is still 2.8.0
- hive is upgraded to 2.1.1
- the rest of the dependency changes are required for the hive version bump.

There is are a few changes in the codebase, but of course no new functionality 
at all:
- in the TestParquetImport class, the new implementation returns a Utf8 object 
for Strings written out.
- Added the security policy and related code changes from the patch for 
SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
- modified HiveMiniCluster config so it won't try to start a web ui (it's 
unnecessary during tests anyway)


Diffs (updated)
-

  build.gradle fc7fc0c4 
  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
  src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
  testdata/hcatalog/conf/hive-site.xml 8a84a5d3 


Diff: https://reviews.apache.org/r/68687/diff/5/

Changes: https://reviews.apache.org/r/68687/diff/4-5/


Testing
---

Ant unit and 3rd party tests were successful.
gradlew test and thirdpartytest were succesful as well.


Thanks,

Fero Szabo

[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650311#comment-16650311
 ] 

Fero Szabo commented on SQOOP-3381:
---

Hi [~dvoros],

Thanks for letting me know.

Anyway, I've just updated my patch on Reviewboard. I encountered the same 
security policy related issue as you did with the Hadoop upgrade in SQOOP-3305, 
so I've incorporated the DerbyPolicy and the related code changes in mine. I 
hope you approve, and can have a look at it. :)

I've decided to go for an older version of Hive, 2.1.1, since that suffices for 
this parquet upgrade as well.

 

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Oct. 15, 2018, 2:13 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description (updated)
---

This change upgrades our parquet library to the newest version and a whole lot 
of libraries to newer versions with it.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

A few things to note:
- hadoop's version is still 2.8.0
- hive is upgraded to 2.1.1
- the rest of the dependency changes are required for the hive version bump.

There is are a few changes in the codebase, but of course no new functionality 
at all:
- in the TestParquetImport class, the new implementation returns a Utf8 object 
for Strings written out.
- Added the security policy and related code changes from the patch for 
SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
- modified HiveMiniCluster config so it won't try to start a web ui (it's 
unnecessary during tests anyway)


Diffs (updated)
-

  build.gradle fc7fc0c4 
  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
  src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
  testdata/hcatalog/conf/hive-site.xml 8a84a5d3 


Diff: https://reviews.apache.org/r/68687/diff/4/

Changes: https://reviews.apache.org/r/68687/diff/3-4/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test and thirdpartytest were succesful as well.


Thanks,

Fero Szabo

Re: Review Request 68979: SQOOP-3384: Document import into external Hive table backed by S3

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68979/#review209540
---


Ship it!




Ship It!

- Fero Szabo


On Oct. 10, 2018, 3:42 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68979/
> ---
> 
> (Updated Oct. 10, 2018, 3:42 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3384
> https://issues.apache.org/jira/browse/SQOOP-3384
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Document import into external Hive table backed by S3
> 
> 
> Diffs
> -
> 
>   src/docs/user/s3.txt 3724454d7efda6b390a5984d9be44d20c404f766 
> 
> 
> Diff: https://reviews.apache.org/r/68979/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean docs
> ./gradlew docs
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646753#comment-16646753
 ] 

Fero Szabo commented on SQOOP-2331:
---

Hi [~standon],

I wonder if you've managed to find the time to work on this? Or, can you share 
any detail on when you might be able to?

Thanks,

Fero

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.7
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch, SQOOP-2331_2.patch, SQOOP-2331_3.patch
>
>
> Current Apache Sqoop 1.4.7 does not compress in gzip format with 
>  --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639554#comment-16639554
 ] 

Fero Szabo commented on SQOOP-3381:
---

Hi [~dvoros],

Thanks for your comment and sorry for the late answer!

I've been pretty busy in the last few weeks with other issues, but now am ready 
to continue working on this one. Thanks for pointing me to the shaded 
parquet-hadoop-bundle. A few Hive tests are failing because of it... It makes 
me wonder if you've made progress with SQOOP-3305 in the meantime, and if 
upgrading Hive to 3.1.0 would solve this problem?

Because of the failing Hive tests, I haven't tested on a cluster, yet, but will 
certainly do so. 

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68712: SQOOP-3376: Test import into external Hive table backed by S3

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68712/#review209226
---


Ship it!




Lgtm!

It is interesting to see that you ran into the problem that parameterized tests 
don't support multiple dimensions!

In any case, I like the tests as they are now, they are concise enough and 
descriptive enough.

My only concern is documentation, and that it should also cover the kinks and 
quirks. But I see you've filed a separate Jira for that.


src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java
Lines 100 (patched)
<https://reviews.apache.org/r/68712/#comment293541>

Typo :) e missing from TestCase



src/test/org/apache/sqoop/testutil/S3TestUtils.java
Lines 136-138 (original), 153-155 (patched)
<https://reviews.apache.org/r/68712/#comment293542>

This sounds like something we should also mention in the user guide!


- Fero Szabo


On Sept. 24, 2018, 11:12 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68712/
> ---
> 
> (Updated Sept. 24, 2018, 11:12 p.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3376
> https://issues.apache.org/jira/browse/SQOOP-3376
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Testing the Text and Parquet imports into an external Hive table backed by S3.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/s3/TestS3ExternalHiveTableImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/HiveServer2TestUtil.java 
> 799370816cccda7578d7c64add6e283d3123e1c8 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 0e6ef5bf001797aa70a7ad50d261c6fd384222fe 
> 
> 
> Diff: https://reviews.apache.org/r/68712/diff/3/
> 
> 
> Testing
> ---
> 
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Created] (SQOOP-3386) Add DB2 support to upstream documentation.

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3386:
-

 Summary: Add DB2 support to upstream documentation.
 Key: SQOOP-3386
 URL: https://issues.apache.org/jira/browse/SQOOP-3386
 Project: Sqoop
  Issue Type: Task
Reporter: Fero Szabo


DB2 is actually supported by Sqoop, but is not in the list of supported 
databases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68541: SQOOP-3104: Create test categories instead of test suites and naming conventions

[Fero Szabo]

Hey Natalie,

Kudos, for working on this even on the weekend!

All of us appreciate the effort!

Cheers,
Fero

On Sat, Sep 22, 2018 at 7:01 PM Nguyen Truong 
wrote:

>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68541/
> ---
>
> (Updated Sept. 23, 2018, 2:01 a.m.)
>
>
> Review request for Sqoop.
>
>
> Changes
> ---
>
> Hi all,
> Thank you very much for your review and detailed suggestion.
> Sorry, it took me forever to update this diff.
> Best,
> Natalie
>
>
> Bugs: SQOOP-3104
> https://issues.apache.org/jira/browse/SQOOP-3104
>
>
> Repository: sqoop-trunk
>
>
> Description
> ---
>
> We are currently unsing test naming conventions to differentiate between
> ManualTests, Unit tests and 3rd party tests. Instead of that, I implemented
> junit categories which will allow us to have more categories in the future.
> This would also remove the reliance on the test class name.
>
> Test categories skeleton:
>   SqoopTest _ UnitTest
>   |__ IntegrationTest
>   |__ ManualTest
>
>   ThirdPartyTest _ CubridTest
>|__ Db2Test
>|__ MainFrameTest
>|__ MysqlTest
>|__ NetezzaTest
>|__ OracleTest
>|__ PostgresqlTest
>|__ SqlServerTest
>
>   KerberizedTest
>
> Categories explanation:
> * SqoopTest: Group of the big categories, including:
> - UnitTest: It tests one class only with its dependencies mocked
> or if the dependency
> is lightweight we can keep it. It must not start a minicluster or
> an hsqldb database.
> It does not need JCDB drivers.
> - IntegrationTest: It usually tests a whole scenario. It may start
> up miniclusters,
> hsqldb and connect to external resources like RDBMSs.
> - ManualTest: This should be a deprecated category which should
> not be used in the future.
> It only exists to mark the currently existing manual tests.
> * ThirdPartyTest: An orthogonal hierarchy for tests that need a JDBC
> driver and/or a docker
> container/external RDBMS instance to run. Subcategories express what
> kind of external
> resource the test needs. E.g: OracleTest needs an Oracle RDBMS and
> Oracle driver on the classpath
> * KerberizedTest: Test that needs Kerberos, which needs to be run on a
> separate JVM.
>
> Opinions are very welcomed. Thanks!
>
>
> Diffs (updated)
> -
>
>   build.gradle fc7fc0c4c
>   src/test/org/apache/sqoop/TestConnFactory.java fb6c94059
>   src/test/org/apache/sqoop/TestIncrementalImport.java 29c477954
>   src/test/org/apache/sqoop/TestSqoopOptions.java e55682edf
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java 631eeff5e
>   src/test/org/apache/sqoop/authentication/TestKerberosAuthenticator.java
> f5700ce65
>   src/test/org/apache/sqoop/db/TestDriverManagerJdbcConnectionFactory.java
> 244831672
>
> src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
> d3e3fb23e
>   src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java c4caafba5
>   src/test/org/apache/sqoop/hbase/HBaseKerberizedConnectivityTest.java
> 3bfb39178
>   src/test/org/apache/sqoop/hbase/HBaseUtilTest.java c6a808c33
>   src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java e78a535f4
>   src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java ba05cabbb
>
> src/test/org/apache/sqoop/hive/HiveServer2ConnectionFactoryInitializerTest.java
> 4d2cb2f88
>   src/test/org/apache/sqoop/hive/TestHiveClientFactory.java a3c2dc939
>   src/test/org/apache/sqoop/hive/TestHiveMiniCluster.java 419f888c0
>   src/test/org/apache/sqoop/hive/TestHiveServer2Client.java 02617295e
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java
> b55179a4f
>   src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java 410724f37
>   src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java
> 276e9eaa4
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java 626ad22f6
>   src/test/org/apache/sqoop/hive/TestTableDefWriterForExternalTable.java
> f1768ee76
>
> src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java
> ff13dc3bc
>   src/test/org/apache/sqoop/io/TestCodecMap.java e71921823
>   src/test/org/apache/sqoop/io/TestLobFile.java 2bc95f283
>   src/test/org/apache/sqoop/io/TestNamedFifo.java a93784e08
>   src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java c59aa26ad
>   src/test/org/apache/sqoop/lib/TestBlobRef.java b271d3c7b
>   src/test/org/apache/sqoop/lib/TestBooleanParser.java 914ab37e4
>   src/test/org/apache/sqoop/lib/TestClobRef.java f94d1a8af
>   src/test/org/apache/sqoop/lib/TestFieldFormatter.java 9ac55e703
>   src/test/org/apache/sqoop/lib/TestLargeObjectLoader.java 1e07d7174
>   

Re: [sqoop-2639] patch for sqoop-2639

[Fero Szabo]

Hi,

Can you please link the review in the Jira?
(Or at least on this thread?)

Thanks,
Fero

On Fri, Sep 14, 2018 at 8:39 AM, CharSyam  wrote:

> Thanks Fero. I created review in ReviewBoard.
>
> 2018년 9월 14일 (금) 오후 8:34, Fero Szabo 님이 작성:
>
> > Hi,
> >
> > Thank you for your contribution!
> >
> > Since the Sqoop community is using ReviewBoard for reviews, the next step
> > would be to create a review there.
> >
> > This is how I explained this process on another Jira:
> > Please go to Review Board at https://reviews.apache.org/account/login/
> and
> > register if you haven't done so far. Then, create a patch by invoking
> *git
> > diff > SQOOP-2949
> > <https://issues.apache.org/jira/browse/SQOOP-2949>-1.patch on the
> command
> > line. Finally, create a review using the sqoop-trunk* repository and your
> > patch. Fill in the necessary fields, as for example, in this review:
> > https://reviews.apache.org/r/65607/  (no need for a description this
> long,
> > nobody likes to read this much   ).
> >
> > After your patch has been reviewed, hopefully, a committer or PMC will
> pick
> > it up and commit it.
> >
> > Best Regards,
> > Fero
> >
> >
> > On Fri, Sep 14, 2018 at 1:06 PM, CharSyam  wrote:
> >
> > > you can test adding --mysql-charset UTF-8 options in commandline
> > >
> > > 2018년 9월 14일 (금) 오후 8:05, CharSyam 님이 작성:
> > >
> > > > Hi, I made a patch for sqoop-2639
> > > > actually this is for 1.4.6 branch, I also made for trunk branch.
> > > >
> > > > How I can upload this patch or get review?
> > > >
> > > > Thanks. all
> > > >
> > >
> >
> >
> >
> > --
> > *Ferenc Szabo* | Software Engineer
> > t. (+361) 701 1201 <+361+701+1201>
> > cloudera.com <https://www.cloudera.com>
> >
> > [image: Cloudera] <https://www.cloudera.com/>
> >
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > --
> >
>



-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com <https://www.cloudera.com>

[image: Cloudera] <https://www.cloudera.com/>

[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
--

[jira] [Commented] (SQOOP-2639) Unable to export utf-8 data to MySQL using --direct mode

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618300#comment-16618300
 ] 

Fero Szabo commented on SQOOP-2639:
---

Hi [~charsyam],

I presume you want to contribute to Sqoop? ;)

I cannot grant you any privileges, since I'm just a contributor myself, but in 
this case, you could ask to be added as contributor on the Sqoop-dev mailing 
list.

Best Regards,

Fero

> Unable to export utf-8 data to MySQL using --direct mode
> 
>
> Key: SQOOP-2639
> URL: https://issues.apache.org/jira/browse/SQOOP-2639
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql
>Affects Versions: 1.4.6
>Reporter: Ranjan Bagchi
>Priority: Major
> Attachments: sqoop-2639.patch
>
>
> I am able to import utf-8 data (non-latin1) data successfully into HDFS via:
> sqoop import --connect jdbc:mysql://host/db --username XX --password YY \
> --mysql-delimiters \
> --table MYSQL_SRC_TABLE --target-dir ${SQOOP_DIR_PREFIX}/mysql_table 
> --direct 
> However, using 
> sqoop export --connect  jdbc:mysql://host/db --username XX --password YY \
> --mysql-delimiters \
> --table MYSQL_DEST_TABLE --export-dir ${SQOOP_DIR_PREFIX}/mysql_table 
> \
> --direct 
> Cuts off the fields after the first non-latin1 character (eg a letter w/ an 
> umlaut).
> I tried other options like  -- --default-character-set=utf8, without success.
> I was able to fix the problem with the following change:
> Change 
> https://svn.apache.org/repos/asf/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/MySQLExportMapper.java,
>  line 322 from 
> this.mysqlCharSet = MySQLUtils.MYSQL_DEFAULT_CHARSET;
> to
> this.mysqlCharSet = "utf-8"; 
> Hope this helps



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [sqoop-2639] patch for sqoop-2639

[Fero Szabo]

Hi,

Thank you for your contribution!

Since the Sqoop community is using ReviewBoard for reviews, the next step
would be to create a review there.

This is how I explained this process on another Jira:
Please go to Review Board at https://reviews.apache.org/account/login/ and
register if you haven't done so far. Then, create a patch by invoking *git
diff > SQOOP-2949
-1.patch on the command
line. Finally, create a review using the sqoop-trunk* repository and your
patch. Fill in the necessary fields, as for example, in this review:
https://reviews.apache.org/r/65607/  (no need for a description this long,
nobody likes to read this much   ).

After your patch has been reviewed, hopefully, a committer or PMC will pick
it up and commit it.

Best Regards,
Fero


On Fri, Sep 14, 2018 at 1:06 PM, CharSyam  wrote:

> you can test adding --mysql-charset UTF-8 options in commandline
>
> 2018년 9월 14일 (금) 오후 8:05, CharSyam 님이 작성:
>
> > Hi, I made a patch for sqoop-2639
> > actually this is for 1.4.6 branch, I also made for trunk branch.
> >
> > How I can upload this patch or get review?
> >
> > Thanks. all
> >
>



-- 
*Ferenc Szabo* | Software Engineer
t. (+361) 701 1201 <+361+701+1201>
cloudera.com 

[image: Cloudera] 

[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 
--

[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612271#comment-16612271
 ] 

Fero Szabo commented on SQOOP-3381:
---

Hi [~dvoros],

I think this change might affect the Hadoop 3.0 upgrade...

Can you perhaps comment on this? 

Thanks!

Fero

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3381:
--
Summary: Upgrade the Parquet library from 1.6.0 to 1.9.0  (was: Upgrade the 
Parquet library)

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:53 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest was successful as well. (it had 1 failed test at first 
run: HBaseImportTest#testAppendWithTimestampSucceeds)


Thanks,

Fero Szabo

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:15 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest had 1 failed test: 
HBaseImportTest#testAppendWithTimestampSucceeds

This seems to be unrealted.


Thanks,

Fero Szabo

Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/1/


Testing
---

Ant unit and 3rd party tests were successful.


Thanks,

Fero Szabo

[jira] [Created] (SQOOP-3382) Reafactor existing Avro numeric tests for reusability (with Parquet)

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3382:
-

 Summary: Reafactor existing Avro numeric tests for reusability 
(with Parquet)
 Key: SQOOP-3382
 URL: https://issues.apache.org/jira/browse/SQOOP-3382
 Project: Sqoop
  Issue Type: Sub-task
Affects Versions: 1.4.7
Reporter: Fero Szabo
Assignee: Fero Szabo
 Fix For: 3.0.0


The current Avro numeric tests are suitable to be used as Parquet tests, with 
very minor modifications, as parquet can be written with the same input and 
nearly the same args. Since we are writing Parquet with it's Avro support, it 
would be good to cover this code with the same, or similar tests (including the 
edge cases related to padding, missing scale and precision cases).

Differences are:
 * the expected output, since stored in a parquet file is different.
 * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3381) Upgrade the Parquet library

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3381:
-

 Summary: Upgrade the Parquet library
 Key: SQOOP-3381
 URL: https://issues.apache.org/jira/browse/SQOOP-3381
 Project: Sqoop
  Issue Type: Sub-task
Affects Versions: 1.4.7
Reporter: Fero Szabo
Assignee: Fero Szabo
 Fix For: 3.0.0


As we will need to register a data supplier in the fix for parquet decimal 
support, we will need a version that contains PARQUET-243.

We need to upgrade the Parquet library to a version that contains this fix and 
is compatible with Hadoop. Most probably, the newest version will be adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3380) parquet-configurator-implementation is not recognized as an option

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3380:
-

 Summary: parquet-configurator-implementation is not recognized as 
an option
 Key: SQOOP-3380
 URL: https://issues.apache.org/jira/browse/SQOOP-3380
 Project: Sqoop
  Issue Type: Bug
Reporter: Fero Szabo
Assignee: Szabolcs Vasas


The parquet-configurator-implementation option was added to Sqoop with 
SQOOP-3329: Remove Kite dependency from the Sqoop project, but the command line 
parser doesn't recognize it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2949) SQL Syntax error when split-by column is of character type and min or max value has single quote inside it

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601955#comment-16601955
 ] 

Fero Szabo commented on SQOOP-2949:
---

Hi [~gireeshp],

My email is [f...@cloudera.com|mailto:f...@cloudera.com] 

The release process doesn't have a defined schedule, yet, so there is no 
timeline. There is only 1 item left from the discussed items that is still 
pending (Hadoop 3 / Hive 3 / Hbase 2 support), i.e. just a library upgrade on 
the Sqoop side.

> SQL Syntax error when split-by column is of character type and min or max 
> value has single quote inside it
> --
>
> Key: SQOOP-2949
> URL: https://issues.apache.org/jira/browse/SQOOP-2949
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Sqoop 1.4.6
> Run on Hadoop 2.6.0
> On Ubuntu
>Reporter: Gireesh Puthumana
>Assignee: Gireesh Puthumana
>Priority: Major
>
> Did a sqoop import from mysql table "emp", with split-by column "ename", 
> which is a varchar(100) type.
> +Used below command:+
> sqoop import --connect jdbc:mysql://localhost/testdb --username root 
> --password * --table emp --m 2 --target-dir /sqoopTest/5 --split-by ename;
> +Ename has following records:+
> | ename   |
> | gireesh |
> | aavesh  |
> | shiva'  |
> | jamir   |
> | balu|
> | santosh |
> | sameer  |
> Min value is "aavesh" and max value is "shiva'" (please note the single quote 
> inside max value).
> When run, it tried to execute below query in mapper 2 and failed:
> SELECT `ename`, `eid`, `deptid` FROM `emp` AS `emp` WHERE ( `ename` >= 
> 'jd聯聭聪G耀' ) AND ( `ename` <= 'shiva'' )
> +Stack trace:+
> {quote}
> 2016-06-05 16:54:06,749 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error 
> in your SQL syntax; check the manual that corresponds to your MySQL server 
> version for the right syntax to use near ''shiva'' )' at line 1
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>   at com.mysql.jdbc.Util.getInstance(Util.java:387)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:942)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>   at 
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68569: HiveMiniCluster does not restore hive-site.xml location

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68569/#review208248
---


Ship it!




Ship It!

- Fero Szabo


On Aug. 30, 2018, 11:27 a.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68569/
> ---
> 
> (Updated Aug. 30, 2018, 11:27 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3375
> https://issues.apache.org/jira/browse/SQOOP-3375
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> HiveMiniCluster sets the hive-site.xml location using 
> org.apache.hadoop.hive.conf.HiveConf#setHiveSiteLocation static method during 
> startup but it does not restore the original location during shutdown.
> 
> This makes HCatalogImportTest and HCatalogExportTest fail if they are ran in 
> the same JVM after any test using HiveMiniCluster.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 19bb7605c 
> 
> 
> Diff: https://reviews.apache.org/r/68569/diff/1/
> 
> 
> Testing
> ---
> 
> Executed unit and third party tests.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

[jira] [Updated] (SQOOP-3377) True Parquet Decimal Support

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3377:
--
Description: 
Currently, fixed point numbers (decimal, number) are stored as a String when 
imported to parquet. This Jira is about adding the capability to store them as 
logical types (as we do in avro).

The parquet library might have to be upgraded.

  was:Currently, fixed point numbers (decimal, number) are stored as a String 
when imported to parquet. This Jira is about adding the capability to store 
them as logical types (as we do in avro).


> True Parquet Decimal Support
> 
>
> Key: SQOOP-3377
> URL: https://issues.apache.org/jira/browse/SQOOP-3377
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>    Reporter: Fero Szabo
>    Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> Currently, fixed point numbers (decimal, number) are stored as a String when 
> imported to parquet. This Jira is about adding the capability to store them 
> as logical types (as we do in avro).
> The parquet library might have to be upgraded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3377) True Parquet Decimal Support

[Fero Szabo (JIRA)]

Fero Szabo created SQOOP-3377:
-

 Summary: True Parquet Decimal Support
 Key: SQOOP-3377
 URL: https://issues.apache.org/jira/browse/SQOOP-3377
 Project: Sqoop
  Issue Type: Improvement
Affects Versions: 1.4.7
Reporter: Fero Szabo
Assignee: Fero Szabo
 Fix For: 3.0.0


Currently, fixed point numbers (decimal, number) are stored as a String when 
imported to parquet. This Jira is about adding the capability to store them as 
logical types (as we do in avro).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598603#comment-16598603
 ] 

Fero Szabo commented on SQOOP-2331:
---

[~standon],

Also, [~BoglarkaEgyed] mentioned it to me that Sqoop has a test that you might 
be able to reuse:

org.apache.sqoop.TestCompression

 

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.7
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch, SQOOP-2331_2.patch, SQOOP-2331_3.patch
>
>
> Current Apache Sqoop 1.4.7 does not compress in gzip format with 
>  --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2949) SQL Syntax error when split-by column is of character type and min or max value has single quote inside it

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597566#comment-16597566
 ] 

Fero Szabo commented on SQOOP-2949:
---

Hi [~gireeshp],

Do you have an update on this issue? We'd like to get this one into the next 
release, if possible... :)

> SQL Syntax error when split-by column is of character type and min or max 
> value has single quote inside it
> --
>
> Key: SQOOP-2949
> URL: https://issues.apache.org/jira/browse/SQOOP-2949
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Sqoop 1.4.6
> Run on Hadoop 2.6.0
> On Ubuntu
>Reporter: Gireesh Puthumana
>Assignee: Gireesh Puthumana
>Priority: Major
>
> Did a sqoop import from mysql table "emp", with split-by column "ename", 
> which is a varchar(100) type.
> +Used below command:+
> sqoop import --connect jdbc:mysql://localhost/testdb --username root 
> --password * --table emp --m 2 --target-dir /sqoopTest/5 --split-by ename;
> +Ename has following records:+
> | ename   |
> | gireesh |
> | aavesh  |
> | shiva'  |
> | jamir   |
> | balu|
> | santosh |
> | sameer  |
> Min value is "aavesh" and max value is "shiva'" (please note the single quote 
> inside max value).
> When run, it tried to execute below query in mapper 2 and failed:
> SELECT `ename`, `eid`, `deptid` FROM `emp` AS `emp` WHERE ( `ename` >= 
> 'jd聯聭聪G耀' ) AND ( `ename` <= 'shiva'' )
> +Stack trace:+
> {quote}
> 2016-06-05 16:54:06,749 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error 
> in your SQL syntax; check the manual that corresponds to your MySQL server 
> version for the right syntax to use near ''shiva'' )' at line 1
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>   at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>   at com.mysql.jdbc.Util.getInstance(Util.java:387)
>   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:942)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
>   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
>   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
>   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
>   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
>   at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
>   at 
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594979#comment-16594979
 ] 

Fero Szabo commented on SQOOP-2331:
---

Hi [~standon],

I've reviewed the modified code and made a few suggestions on ReviewBoard. Can 
you please have a look? 

Also, I wonder if you saw the update there when I posted it, as it was two 
weeks ago... 

Thanks,

Fero

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.7
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch, SQOOP-2331_2.patch, SQOOP-2331_3.patch
>
>
> Current Apache Sqoop 1.4.7 does not compress in gzip format with 
>  --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

[Fero Szabo via Review Board]

> On Aug. 28, 2018, 11:41 a.m., Fero Szabo wrote:
> >

All ant tests passed on my system. (unit, 3rd party and s3). Gradle unit tests 
passed as well.


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/#review208030
---


On Aug. 28, 2018, 8:33 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68475/
> ---
> 
> (Updated Aug. 28, 2018, 8:33 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3363
> https://issues.apache.org/jira/browse/SQOOP-3363
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> * Added new test cases for Parquet import into S3 as it was still missing
> * Added new test cases for incremental append import into S3 in Text, Avro, 
> Sequence and Parquet file format
> * Added new test cases for incremental merge import into S3 in Text and 
> Parquet file format
> * Updated some previously added logic in S3 util and test classes
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java 
> 20c0d13c391d312a943a147882cf85f86cb7e438 
>   src/java/org/apache/sqoop/util/FileSystemUtil.java 
> 96ec21233d6378865195008f5ab67a74370fa2ed 
>   src/test/org/apache/sqoop/TestAppendUtils.java 
> 3d66beca0226dd9eb8a14f4157444f24a247491a 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> e130c42104b86e854d45babc009a5f1409a74a48 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> c17c1c54918df0b4d1ecbaef4e381975d72756ae 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
>   src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
> 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
> ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
>   src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
> ad7576dbb2447423c677429f24163031a9d39b5f 
>   src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
> df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 
> 
> 
> Diff: https://reviews.apache.org/r/68475/diff/5/
> 
> 
> Testing
> ---
> 
> ant clean test -Ds3.bucket.url= 
> -Ds3.generator.command=
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>