[jira] [Commented] (SQOOP-3380) parquet-configurator-implementation is not recognized as an option
[ https://issues.apache.org/jira/browse/SQOOP-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611500#comment-16611500 ] Hudson commented on SQOOP-3380: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1207 (See [https://builds.apache.org/job/Sqoop-hadoop200/1207/]) SQOOP-3380: parquet-configurator-implementation is not recognized as an (vasas: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=b37e0c6053b92f2d7a56026e7fde5637b7f70588]) * (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java * (edit) src/test/org/apache/sqoop/tool/TestBaseSqoopTool.java > parquet-configurator-implementation is not recognized as an option > -- > > Key: SQOOP-3380 > URL: https://issues.apache.org/jira/browse/SQOOP-3380 > Project: Sqoop > Issue Type: Bug >Reporter: Fero Szabo >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 3.0.0 > > Attachments: SQOOP-3380.patch > > > The parquet-configurator-implementation option was added to Sqoop with > SQOOP-3329: Remove Kite dependency from the Sqoop project, but the command > line parser doesn't recognize it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3380) parquet-configurator-implementation is not recognized as an option
[ https://issues.apache.org/jira/browse/SQOOP-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611481#comment-16611481 ] ASF subversion and git services commented on SQOOP-3380: Commit b37e0c6053b92f2d7a56026e7fde5637b7f70588 in sqoop's branch refs/heads/trunk from [~vasas] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=b37e0c6 ] SQOOP-3380: parquet-configurator-implementation is not recognized as an option (Szabolcs Vasas) > parquet-configurator-implementation is not recognized as an option > -- > > Key: SQOOP-3380 > URL: https://issues.apache.org/jira/browse/SQOOP-3380 > Project: Sqoop > Issue Type: Bug >Reporter: Fero Szabo >Assignee: Szabolcs Vasas >Priority: Major > Attachments: SQOOP-3380.patch > > > The parquet-configurator-implementation option was added to Sqoop with > SQOOP-3329: Remove Kite dependency from the Sqoop project, but the command > line parser doesn't recognize it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68677: SQOOP-3383: Eliminate duplicated steps in S3 tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68677/ --- (Updated Sept. 11, 2018, 4:34 p.m.) Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas. Changes --- Disable FileSystem's static cache for S3 tests Bugs: SQOOP-3383 https://issues.apache.org/jira/browse/SQOOP-3383 Repository: sqoop-trunk Description --- In the current S3 test suite the setup phase includes a logic which sets the AWS credentials in the Hadoop configuration in the org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method. These credentials then are set via system properties calling the org.apache.sqoop.testutil.S3TestUtils#getArgumentArrayBuilderForS3UnitTests method too. This is an unnecessary duplication that should be cleaned up to ease further test implementation. Diffs (updated) - src/test/org/apache/sqoop/testutil/S3TestUtils.java 7724026b0bad25a31aa76c89135a51538b46bf82 Diff: https://reviews.apache.org/r/68677/diff/2/ Changes: https://reviews.apache.org/r/68677/diff/1-2/ Testing (updated) --- ./gradlew test -Ds3.bucket.url= -Ds3.generator.command= Thanks, Boglarka Egyed
Review Request 68687: SQOOP-3381 Upgrade the Parquet library
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68687/ --- Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas. Bugs: SQOOP-3381 https://issues.apache.org/jira/browse/SQOOP-3381 Repository: sqoop-trunk Description --- This change upgrades our parquet library to the newest version. As we will need to register a data supplier in the fix for parquet decimal support (SQOOP-3382), we will need a version that contains PARQUET-243. We need to upgrade the Parquet library to a version that contains this fix and is compatible with Hadoop 3.0. The only difference is found in the TestParquetImport class, the new implementation returns a Utf8 object for Strings written out. Diffs - gradle.properties 0d30378d gradle/sqoop-package.gradle 1a8d994d ivy.xml 670cb32d ivy/libraries.properties 8f3dab2b src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java 2180cc20 src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java 90b910a3 src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java 66ebc5b8 src/test/org/apache/sqoop/TestParquetExport.java be1d8164 src/test/org/apache/sqoop/TestParquetImport.java 2810e318 src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 Diff: https://reviews.apache.org/r/68687/diff/1/ Testing --- Ant unit and 3rd party tests were successful. Thanks, Fero Szabo
Re: Review Request 68626: parquet-configurator-implementation is not recognized as an option
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68626/#review208515 --- Ship it! Ran unit and 3rd party tests successfully. Thanks for this fix, Szabi! - Boglarka Egyed On Sept. 5, 2018, 2:43 a.m., Szabolcs Vasas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68626/ > --- > > (Updated Sept. 5, 2018, 2:43 a.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3380 > https://issues.apache.org/jira/browse/SQOOP-3380 > > > Repository: sqoop-trunk > > > Description > --- > > The parquet-configurator-implementation option was added to Sqoop with > SQOOP-3329: Remove Kite dependency from the Sqoop project, but the command > line parser doesn't recognize it. > > > Diffs > - > > src/java/org/apache/sqoop/tool/BaseSqoopTool.java b47be72cd > src/test/org/apache/sqoop/tool/TestBaseSqoopTool.java 5571b25a1 > > > Diff: https://reviews.apache.org/r/68626/diff/1/ > > > Testing > --- > > Added 2 new unit test cases, ran unit and third party tests successfully. > > > Thanks, > > Szabolcs Vasas > >
Speakers needed for Apache DC Roadshow
We need your help to make the Apache Washington DC Roadshow on Dec 4th a success. What do we need most? Speakers! We're bringing a unique DC flavor to this event by mixing Open Source Software with talks about Apache projects as well as OSS CyberSecurity, OSS in Government and and OSS Career advice. Please take a look at: http://www.apachecon.com/usroadshow18/ (Note: You are receiving this message because you are subscribed to one or more mailing lists at The Apache Software Foundation.) Rich, for the ApacheCon Planners -- rbo...@apache.org http://apachecon.com @ApacheCon
Re: Review Request 68677: SQOOP-3383: Eliminate duplicated steps in S3 tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68677/#review208509 --- Hi Bogi, I have taken a look at the patch and I think that this duplication is unfortunately needed because the Sqoop import does not pick up the Hadoop configuration which is created at the beginning of the test. It is misleading that the tests are still passing but that happens only because FileSystem has a static cache, so once the authentication happens in the setup() method the Sqoop import just gets the same FileSystem object from the cache so it does not have to authenticate again. (see: org.apache.hadoop.fs.FileSystem#get(java.net.URI, org.apache.hadoop.conf.Configuration)). I think we should rewrite this patch to set fs.s3a.impl.disable.cache property to true on the Hadoop conf created in the setup() to make sure Sqoop will rely on the S3 parameters specified in the site.xml or in -D properties. - Szabolcs Vasas On Sept. 10, 2018, 1:22 p.m., Boglarka Egyed wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68677/ > --- > > (Updated Sept. 10, 2018, 1:22 p.m.) > > > Review request for Sqoop, daniel voros, Fero Szabo, and Szabolcs Vasas. > > > Bugs: SQOOP-3383 > https://issues.apache.org/jira/browse/SQOOP-3383 > > > Repository: sqoop-trunk > > > Description > --- > > In the current S3 test suite the setup phase includes a logic which sets the > AWS credentials in the Hadoop configuration in the > org.apache.sqoop.testutil.S3TestUtils#setS3CredentialsInHadoopConf method. > These credentials then are set via system properties calling the > org.apache.sqoop.testutil.S3TestUtils#getArgumentArrayBuilderForS3UnitTests > method too. > > This is an unnecessary duplication that should be cleaned up to ease further > test implementation. > > > Diffs > - > > src/test/org/apache/sqoop/s3/TestS3AvroImport.java > 7f5f5d62c5cab10f932aa22c3a713b13fefc2b58 > src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java > 5faf59ea80c48fe025294cabd100e7d176032138 > src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java > a4f986423ea299716a29f9d02f7c8453a7f2ba02 > src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java > d271588c5af060bbc3d301a845f45c46d0f6a2ba > src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java > 52d89c775b5f1219471df44d222fd92a59ed408c > src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java > 39238c5fab56b54a85dde5aed0d4bb2c77382fa6 > src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java > 597e3def2cc33adebeeb3bc1ee35ad8a7f4b990d > src/test/org/apache/sqoop/s3/TestS3ParquetImport.java > c9785d816d4a7a5870d74c51a9faa229f6d3818e > src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java > bba8b74ebe639df26e977abf377f4904144dcfaa > src/test/org/apache/sqoop/s3/TestS3TextImport.java > 114f97cbb8857a7633cae5d030769ac4a90e36aa > src/test/org/apache/sqoop/testutil/S3TestUtils.java > 7724026b0bad25a31aa76c89135a51538b46bf82 > > > Diff: https://reviews.apache.org/r/68677/diff/1/ > > > Testing > --- > > ant clean test -Ds3.bucket.url= > -Ds3.generator.command= > > > Thanks, > > Boglarka Egyed > >
