[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Daniel Voros (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612402#comment-16612402
 ] 

Daniel Voros commented on SQOOP-3381:
-

Hey [~fero], thanks for keeping that in mind. What I've seen during the hadoop3 
upgrade, is that Avro is added to the MR classpath from under hadoop. So where 
this could lead to issues is conflicting versions of Avro in hadoop and Parquet 
shipped with Sqoop.

Could you try your patch (having new parquet jar in lib/) on a cluster with 
current Hadoop versions? I don't think we should bother with testing with 
Hadoop 3, we'll face that in the Hadoop 3 patch.

(One more thing to keep in mind, is that parquet-hadoop-bundle is also shaded 
into the hive-exec artifact. However, I think the classes involved in 
PARQUET-243 are not bundled there.)

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612271#comment-16612271
 ] 

Fero Szabo commented on SQOOP-3381:
---

Hi [~dvoros],

I think this change might affect the Hadoop 3.0 upgrade...

Can you perhaps comment on this? 

Thanks!

Fero

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0

[Fero Szabo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fero Szabo updated SQOOP-3381:
--
Summary: Upgrade the Parquet library from 1.6.0 to 1.9.0  (was: Upgrade the 
Parquet library)

> Upgrade the Parquet library from 1.6.0 to 1.9.0
> ---
>
> Key: SQOOP-3381
> URL: https://issues.apache.org/jira/browse/SQOOP-3381
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> As we will need to register a data supplier in the fix for parquet decimal 
> support, we will need a version that contains PARQUET-243.
> We need to upgrade the Parquet library to a version that contains this fix 
> and is compatible with Hadoop. Most probably, the newest version will be 
> adequate. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:53 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest was successful as well. (it had 1 failed test at first 
run: HBaseImportTest#testAppendWithTimestampSucceeds)


Thanks,

Fero Szabo

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:15 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest had 1 failed test: 
HBaseImportTest#testAppendWithTimestampSucceeds

This seems to be unrealted.


Thanks,

Fero Szabo

Re: Review Request 67408: SQOOP-3326: Mainframe FTP listing for GDG should filter out non-GDG datasets in a heterogeneous listing

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67408/#review208552
---



Hi Chris,

Thank you for submitting this patch, I have successfully executed both the unit 
and third party test pack.
My understanding of the GDG datasets is limited can you please explain a bit 
more what the problem is this patch solves? You could give an example which did 
not work properly previously but works correctly with this patch.

I have left some code style related suggestions please take a look.
I can see a big improvement since your last patches, I think you follow much 
more best practices now, which is great! So keep up the good work! :)


src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java
Line 34 (original), 34 (patched)


Just to avoid future refactoring traps can you put 
MainframeFTPFileGdgEntryParser.class.getName() here instead of the string 
constant?



src/java/org/apache/sqoop/mapreduce/mainframe/MainframeFTPFileGdgEntryParser.java
Lines 54 (patched)


It seems that this constructor is never used, do we need it?
I am not sure it would make sense instantiating a 
MainframeFTPFileGdgEntryParser with a different regexp.



src/java/org/apache/sqoop/mapreduce/mainframe/MainframeFTPFileGdgEntryParser.java
Lines 80 (patched)


It might be a bit cleaner to extract this string into a constant.



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
Lines 49 (patched)


I think these fields can all be non-static members.
You can convert the setUpBeforClass into a @Before method and initialize 
the parser field as well in that method (so you can remove the initialization 
from the individual test methods).



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
Lines 52 (patched)


You can use the diamond operator here: new ArrayList<>



src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
Lines 84 (patched)


Instead of this little bit obscure for-loop you could use a nice Java 8 
syntax here for counting the parsed entries:

long count = listing.stream()
  .map(parser::parseFTPEntry)
  .filter(Objects::nonNull)
  .count();


- Szabolcs Vasas


On June 1, 2018, 4:46 a.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67408/
> ---
> 
> (Updated June 1, 2018, 4:46 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Mainframe FTP listing for GDG should filter out non-GDG datasets in a 
> heterogeneous listing as it grabs the last file and in the case where there 
> are other datasets mixed in, the latest file may not be the desired dataset.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java 
> 9d6a2fe7 
>   
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeFTPFileGdgEntryParser.java
>  PRE-CREATION 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java 654721e3 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileGdgEntryParser.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java 90a85194 
> 
> 
> Diff: https://reviews.apache.org/r/67408/diff/2/
> 
> 
> Testing
> ---
> 
> Unit tests. Integration testing locally on developer machine.
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Re: Review Request 67407: SQOOP-3327: Mainframe FTP needs to Include "Migrated" datasets when parsing the FTP list

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67407/#review208551
---


Ship it!




Hi Chris,

Thank you for submitting this patch, I have successfully executed the unit and 
third party tests, I am not really familiar with the technical background of 
the mainframe migrated data sets but based on the examples given in the code it 
should be OK.

I have left only one comment regarding the code style.

Regards,
Szabolcs


src/java/org/apache/sqoop/mapreduce/mainframe/MainframeFTPFileEntryParser.java
Lines 117 (patched)


I think this block could be extracted into a separate method just, it would 
make parseFTPEntry more readable.


- Szabolcs Vasas


On June 1, 2018, 11:57 p.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67407/
> ---
> 
> (Updated June 1, 2018, 11:57 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Mainframe FTP needs to Include "Migrated" datasets when parsing the FTP list. 
> Initially, these were excluded out of the regular expression.
> 
> 
> Diffs
> -
> 
>   
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeFTPFileEntryParser.java
>  f0b87868 
>   
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileEntryParser.java
>  eb0f8c00 
> 
> 
> Diff: https://reviews.apache.org/r/67407/diff/4/
> 
> 
> Testing
> ---
> 
> Unit testing.
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

[jira] [Commented] (SQOOP-3383) Disable FileSystem static cache in S3 tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611697#comment-16611697
 ] 

Hudson commented on SQOOP-3383:
---

SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1208 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1208/])
SQOOP-3383: Disable FileSystem static cache in S3 tests (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=932822aa8fdddeb0aff6445d4f585a599ccb0084])
* (edit) src/test/org/apache/sqoop/testutil/S3TestUtils.java


> Disable FileSystem static cache in S3 tests
> ---
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: SQOOP-3383.patch, SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68677: SQOOP-3383: Disable FileSystem static cache in S3 tests

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68677/#review208549
---


Ship it!




Thank you for submitting the fix, let's ship it!

- Szabolcs Vasas


On Sept. 12, 2018, 6:42 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68677/
> ---
> 
> (Updated Sept. 12, 2018, 6:42 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3383
> https://issues.apache.org/jira/browse/SQOOP-3383
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the -Dfs.s3a.access.key and -Dfs.s3a.secret.key properties 
> is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> 
> This static cache should be disabled (by settin fs.s3a.impl.disable.cache to 
> true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
> credentials set via the -D properties.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 7724026b0bad25a31aa76c89135a51538b46bf82 
> 
> 
> Diff: https://reviews.apache.org/r/68677/diff/2/
> 
> 
> Testing
> ---
> 
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-3383) Disable FileSystem static cache in S3 tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611686#comment-16611686
 ] 

ASF subversion and git services commented on SQOOP-3383:


Commit 932822aa8fdddeb0aff6445d4f585a599ccb0084 in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=932822a ]

SQOOP-3383: Disable FileSystem static cache in S3 tests

(Boglarka Egyed via Szabolcs Vasas)


> Disable FileSystem static cache in S3 tests
> ---
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch, SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68677: SQOOP-3383: Disable FileSystem static cache in S3 tests

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68677/
---

(Updated Sept. 12, 2018, 6:42 a.m.)


Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas.


Changes
---

Updated summary


Summary (updated)
-

SQOOP-3383: Disable FileSystem static cache in S3 tests


Bugs: SQOOP-3383
https://issues.apache.org/jira/browse/SQOOP-3383


Repository: sqoop-trunk


Description
---

FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -Dfs.s3a.access.key and -Dfs.s3a.secret.key properties 
is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin fs.s3a.impl.disable.cache to 
true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D properties.


Diffs
-

  src/test/org/apache/sqoop/testutil/S3TestUtils.java 
7724026b0bad25a31aa76c89135a51538b46bf82 


Diff: https://reviews.apache.org/r/68677/diff/2/


Testing
---

./gradlew test -Ds3.bucket.url= 
-Ds3.generator.command=


Thanks,

Boglarka Egyed

[jira] [Updated] (SQOOP-3383) Disable FileSystem static cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Summary: Disable FileSystem static cache in S3 tests  (was: Disable 
fs.s3a.impl.disable.cache in S3 tests)

> Disable FileSystem static cache in S3 tests
> ---
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch, SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3383) Disable fs.s3a.impl.disable.cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Attachment: SQOOP-3383.patch

> Disable fs.s3a.impl.disable.cache in S3 tests
> -
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch, SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68677: SQOOP-3383: Eliminate duplicated steps in S3 tests

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68677/
---

(Updated Sept. 12, 2018, 6:38 a.m.)


Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas.


Changes
---

Updated description


Bugs: SQOOP-3383
https://issues.apache.org/jira/browse/SQOOP-3383


Repository: sqoop-trunk


Description (updated)
---

FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -Dfs.s3a.access.key and -Dfs.s3a.secret.key properties 
is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin fs.s3a.impl.disable.cache to 
true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D properties.


Diffs
-

  src/test/org/apache/sqoop/testutil/S3TestUtils.java 
7724026b0bad25a31aa76c89135a51538b46bf82 


Diff: https://reviews.apache.org/r/68677/diff/2/


Testing
---

./gradlew test -Ds3.bucket.url= 
-Ds3.generator.command=


Thanks,

Boglarka Egyed

[jira] [Updated] (SQOOP-3383) Disable fs.s3a.impl.disable.cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Description: 
FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
properties is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
to true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D properties.

  was:
FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -D{{fs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
properties is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
to true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D properties.


> Disable fs.s3a.impl.disable.cache in S3 tests
> -
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the \{{-Dfs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3383) Disable fs.s3a.impl.disable.cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Description: 
FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -D{{fs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
properties is effectless. See org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
to true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D properties.

  was:
FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -D system properties is effectless. See 
org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
to true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D system properties.


> Disable fs.s3a.impl.disable.cache in S3 tests
> -
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the -D{{fs.s3a.access.key}} and -{{Dfs.s3a.secret.key}} 
> properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3383) Disable fs.s3a.impl.disable.cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Description: 
FileSystem has a static cache meaning when the authentication happens in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method Sqoop 
import will get the same FileSystem object from the cache thus its 
authentication via the -D system properties is effectless. See 
org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
org.apache.hadoop.conf.Configuration). 

This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
to true) in the setup phase of the S3 tests to make sure Sqoop relies on the S3 
credentials set via the -D system properties.

  was:
In the current S3 test suite the setup phase includes a logic which sets the 
AWS credentials in the Hadoop configuration in the 
org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method. 
These credentials then are set via system properties calling the 
org.apache.sqoop.testutil.S3TestUtils#getArgumentArrayBuilderForS3UnitTests 
method too.

This is an unnecessary duplication that should be cleaned up to ease further 
test implementation.


> Disable fs.s3a.impl.disable.cache in S3 tests
> -
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch
>
>
> FileSystem has a static cache meaning when the authentication happens in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method 
> Sqoop import will get the same FileSystem object from the cache thus its 
> authentication via the -D system properties is effectless. See 
> org.apache.hadoop.fs.FileSystem#get(java.net.URI, 
> org.apache.hadoop.conf.Configuration). 
> This static cache should be disabled (by settin {{fs.s3a.impl.disable.cache}} 
> to true) in the setup phase of the S3 tests to make sure Sqoop relies on the 
> S3 credentials set via the -D system properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3383) Disable fs.s3a.impl.disable.cache in S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3383:
--
Summary: Disable fs.s3a.impl.disable.cache in S3 tests  (was: Eliminate 
duplicated steps in S3 tests)

> Disable fs.s3a.impl.disable.cache in S3 tests
> -
>
> Key: SQOOP-3383
> URL: https://issues.apache.org/jira/browse/SQOOP-3383
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3383.patch
>
>
> In the current S3 test suite the setup phase includes a logic which sets the 
> AWS credentials in the Hadoop configuration in the 
> org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method. 
> These credentials then are set via system properties calling the 
> org.apache.sqoop.testutil.S3TestUtils#getArgumentArrayBuilderForS3UnitTests 
> method too.
> This is an unnecessary duplication that should be cleaned up to ease further 
> test implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)