[jira] [Commented] (SQOOP-3104) Create test categories instead of test suites and naming conventions

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686819#comment-16686819
 ] 

Hudson commented on SQOOP-3104:
---

FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1227 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1227/])
SQOOP-3104: Create test categories instead of test suites and naming (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=d58e5f106ba932879112d7b69997301e442335f1])
* (edit) src/test/org/apache/sqoop/manager/sqlserver/SQLServerSplitByTest.java
* (edit) src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java
* (add) 
src/test/org/apache/sqoop/util/BlockJUnit4ClassRunnerWithParametersFactory.java
* (add) 
src/test/org/apache/sqoop/testcategories/thirdpartytest/SqlServerTest.java
* (edit) 
src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeExportSequenceFileTest.java
* (edit) src/test/org/apache/sqoop/manager/sqlserver/SQLServerMultiMapsTest.java
* (edit) src/test/org/apache/sqoop/manager/oracle/OracleCompatTest.java
* (edit) src/test/org/apache/sqoop/manager/oracle/OracleSplitterTest.java
* (edit) src/test/org/apache/sqoop/tool/TestImportTool.java
* (edit) src/test/org/apache/sqoop/manager/oracle/OracleLobAvroImportTest.java
* (add) src/test/org/apache/sqoop/testcategories/thirdpartytest/CubridTest.java
* (edit) src/test/org/apache/sqoop/manager/mysql/MySQLAuthTest.java
* (edit) src/test/org/apache/sqoop/manager/netezza/NetezzaExportManualTest.java
* (edit) 
src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerExportTest.java
* (edit) src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java
* (edit) 
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java
* (edit) src/test/org/apache/sqoop/manager/postgresql/PostgresqlImportTest.java
* (edit) 
src/test/org/apache/sqoop/metastore/sqlserver/SqlServerMetaConnectIncrementalImportTest.java
* (edit) src/test/org/apache/sqoop/tool/TestExportToolValidateOptions.java
* (add) 
src/test/org/apache/sqoop/testcategories/thirdpartytest/MainFrameTest.java
* (edit) src/test/org/apache/sqoop/metastore/postgres/PostgresJobToolTest.java
* (edit) src/test/org/apache/sqoop/manager/oracle/OracleCallExportTest.java
* (edit) src/test/org/apache/sqoop/testutil/TestArgumentArrayBuilder.java
* (edit) src/test/org/apache/sqoop/util/TestOptionsFileExpansion.java
* (edit) src/test/org/apache/sqoop/metastore/db2/DB2JobToolTest.java
* (edit) 
src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java
* (add) src/test/org/apache/sqoop/testcategories/thirdpartytest/NetezzaTest.java
* (edit) src/test/org/apache/sqoop/hive/TestHiveServer2Client.java
* (edit) 
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeFTPFileEntryParser.java
* (edit) 
src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeImportDelimitedFileTest.java
* (edit) src/test/org/apache/sqoop/manager/sqlserver/SQLServerWhereTest.java
* (edit) src/test/org/apache/sqoop/util/TestFileSystemUtil.java
* (edit) 
src/test/org/apache/sqoop/manager/sqlserver/SQLServerParseMethodsTest.java
* (edit) src/test/org/apache/sqoop/metastore/postgres/PostgresSavedJobsTest.java
* (edit) src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
* (edit) src/test/org/apache/sqoop/hive/TestTableDefWriter.java
* (edit) src/test/org/apache/sqoop/orm/TestClassWriter.java
* (add) src/test/org/apache/sqoop/testcategories/thirdpartytest/MysqlTest.java
* (edit) 
src/test/org/apache/sqoop/manager/netezza/DirectNetezzaHCatImportManualTest.java
* (edit) 
src/test/org/apache/sqoop/manager/netezza/DirectNetezzaExportManualTest.java
* (edit) 
src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java
* (edit) 
src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
* (edit) src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java
* (edit) src/test/org/apache/sqoop/metastore/PasswordRedactorTest.java
* (edit) src/test/org/apache/sqoop/io/TestLobFile.java
* (edit) src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java
* (edit) src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java
* (edit) src/test/org/apache/sqoop/util/TestSubstitutionUtils.java
* (edit) src/test/org/apache/sqoop/manager/cubrid/CubridManagerImportTest.java
* (edit) src/test/org/apache/sqoop/validation/AbortOnFailureHandlerTest.java
* (edit) src/test/org/apache/sqoop/hbase/HBaseUtilTest.java
* (edit) src/test/org/apache/sqoop/manager/mysql/DirectMySQLTest.java
* (edit) src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java
* (edit) src/test/org/apache/sqoop/metastore/mysql/MySqlJobToolTest.java
* (edit) src/test/org/apache/sqoop/util/TestDirCleanupHook.java
* (edit) src/test/org/apache/sqoop/manager/cubrid/CubridAuthTest.java
* (add) 
src/test/org/apache/sqoop/testcategories/thirdpartytest/ThirdPartyTest.java
* (edit) 

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Issue Type: Sub-task  (was: Test)
Parent: SQOOP-3228

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> SQOOP-3104 has introduced test categories but while it was under review many 
> other patches with new test cases were committed.
> The task is to make sure that all of the new tests are properly categorized 
> and the test tasks are cleaned up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Description: 
SQOOP-3104 has introduced test categories but while it was under review many 
other patches with new test cases were committed.

The task is to make sure that all of the new tests are properly categorized and 
the test tasks are cleaned up.

  was:
We are currently unsing test naming conventions to differentiate between 
ManualTests, Unit tests and 3rd party tests.

Instead of this, we should use junit4 categories, so it's clear by looking at a 
test where it fits. This would also remove the reliance on the test class name.



> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> SQOOP-3104 has introduced test categories but while it was under review many 
> other patches with new test cases were committed.
> The task is to make sure that all of the new tests are properly categorized 
> and the test tasks are cleaned up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3404:
-

Assignee: Szabolcs Vasas  (was: Nguyen Truong)

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Fix For: 3.0.0
>
>
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests.
> Instead of this, we should use junit4 categories, so it's clear by looking at 
> a test where it fits. This would also remove the reliance on the test class 
> name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3404:
-

 Summary: Categorize all tests in the project
 Key: SQOOP-3404
 URL: https://issues.apache.org/jira/browse/SQOOP-3404
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Szabolcs Vasas
Assignee: Nguyen Truong
 Fix For: 3.0.0


We are currently unsing test naming conventions to differentiate between 
ManualTests, Unit tests and 3rd party tests.

Instead of this, we should use junit4 categories, so it's clear by looking at a 
test where it fits. This would also remove the reliance on the test class name.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3404) Categorize all tests in the project

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3404:
--
Issue Type: Test  (was: Sub-task)
Parent: (was: SQOOP-3228)

> Categorize all tests in the project
> ---
>
> Key: SQOOP-3404
> URL: https://issues.apache.org/jira/browse/SQOOP-3404
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Nguyen Truong
>Priority: Major
> Fix For: 3.0.0
>
>
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests.
> Instead of this, we should use junit4 categories, so it's clear by looking at 
> a test where it fits. This would also remove the reliance on the test class 
> name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3104) Create test categories instead of test suites and naming conventions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686725#comment-16686725
 ] 

ASF subversion and git services commented on SQOOP-3104:


Commit d58e5f106ba932879112d7b69997301e442335f1 in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=d58e5f1 ]

SQOOP-3104: Create test categories instead of test suites and naming conventions

(Nguyen Truong via Szabolcs Vasas)


> Create test categories instead of test suites and naming conventions
> 
>
> Key: SQOOP-3104
> URL: https://issues.apache.org/jira/browse/SQOOP-3104
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Anna Szonyi
>Assignee: Nguyen Truong
>Priority: Major
>
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests.
> Instead of this, we should use junit4 categories, so it's clear by looking at 
> a test where it fits. This would also remove the reliance on the test class 
> name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68541: SQOOP-3104: Create test categories instead of test suites and naming conventions

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68541/#review210545
---


Ship it!




Hi Nguyen,

Thank you very much for the update! Now it looks good to me, let's ship it! :)

Thanks,
Bogi

- Boglarka Egyed


On Nov. 13, 2018, 6:18 a.m., Nguyen Truong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68541/
> ---
> 
> (Updated Nov. 13, 2018, 6:18 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3104
> https://issues.apache.org/jira/browse/SQOOP-3104
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests. Instead of that, I implemented 
> junit categories which will allow us to have more categories in the future. 
> This would also remove the reliance on the test class name.
> 
> Test categories skeleton:
>   SqoopTest _ UnitTest
>   |__ IntegrationTest
>   |__ ManualTest
> 
>   ThirdPartyTest _ CubridTest
>|__ Db2Test
>|__ MainFrameTest
>|__ MysqlTest
>|__ NetezzaTest
>|__ OracleTest
>|__ PostgresqlTest
>|__ SqlServerTest
> 
>   KerberizedTest
> 
> Categories explanation:
> * SqoopTest: Group of the big categories, including:
> - UnitTest: It tests one class only with its dependencies mocked or 
> if the dependency
> is lightweight we can keep it. It must not start a minicluster or an 
> hsqldb database.
> It does not need JCDB drivers.
> - IntegrationTest: It usually tests a whole scenario. It may start up 
> miniclusters,
> hsqldb and connect to external resources like RDBMSs.
> - ManualTest: This should be a deprecated category which should not 
> be used in the future.
> It only exists to mark the currently existing manual tests.
> * ThirdPartyTest: An orthogonal hierarchy for tests that need a JDBC 
> driver and/or a docker
> container/external RDBMS instance to run. Subcategories express what kind 
> of external
> resource the test needs. E.g: OracleTest needs an Oracle RDBMS and Oracle 
> driver on the classpath
> * KerberizedTest: Test that needs Kerberos, which needs to be run on a 
> separate JVM.
> 
> Opinions are very welcomed. Thanks!
> 
> 
> Diffs
> -
> 
>   build.gradle 2014b5cf5 
>   src/test/org/apache/sqoop/TestConnFactory.java fb6c94059 
>   src/test/org/apache/sqoop/TestIncrementalImport.java 29c477954 
>   src/test/org/apache/sqoop/TestSqoopOptions.java e55682edf 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java 631eeff5e 
>   src/test/org/apache/sqoop/authentication/TestKerberosAuthenticator.java 
> f5700ce65 
>   src/test/org/apache/sqoop/db/TestDriverManagerJdbcConnectionFactory.java 
> 244831672 
>   
> src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
>  d3e3fb23e 
>   src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java c4caafba5 
>   src/test/org/apache/sqoop/hbase/HBaseKerberizedConnectivityTest.java 
> 3bfb39178 
>   src/test/org/apache/sqoop/hbase/HBaseTestCase.java 94b71b61c 
>   src/test/org/apache/sqoop/hbase/HBaseUtilTest.java c6a808c33 
>   src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java e78a535f4 
>   src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java ba05cabbb 
>   
> src/test/org/apache/sqoop/hive/HiveServer2ConnectionFactoryInitializerTest.java
>  4d2cb2f88 
>   src/test/org/apache/sqoop/hive/TestHiveClientFactory.java a3c2dc939 
>   src/test/org/apache/sqoop/hive/TestHiveMiniCluster.java 419f888c0 
>   src/test/org/apache/sqoop/hive/TestHiveServer2Client.java 02617295e 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java 65f079467 
>   src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java 410724f37 
>   src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java 
> 276e9eaa4 
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java 626ad22f6 
>   src/test/org/apache/sqoop/hive/TestTableDefWriterForExternalTable.java 
> f1768ee76 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   src/test/org/apache/sqoop/io/TestCodecMap.java e71921823 
>   src/test/org/apache/sqoop/io/TestLobFile.java 2bc95f283 
>   src/test/org/apache/sqoop/io/TestNamedFifo.java a93784e08 
>   src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java c59aa26ad 
>   src/test/org/apache/sqoop/lib/TestBlobRef.java b271d3c7b 
>   src/test/org/apache/sqoop/lib/TestBooleanParser.java 

Re: Review Request 68541: SQOOP-3104: Create test categories instead of test suites and naming conventions

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68541/#review210541
---


Ship it!




Hi Nguyen,

Thank you for updating your patch, I ran the unit and third party tests and 
everything is green.

- Szabolcs Vasas


On Nov. 13, 2018, 6:18 a.m., Nguyen Truong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68541/
> ---
> 
> (Updated Nov. 13, 2018, 6:18 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3104
> https://issues.apache.org/jira/browse/SQOOP-3104
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> We are currently unsing test naming conventions to differentiate between 
> ManualTests, Unit tests and 3rd party tests. Instead of that, I implemented 
> junit categories which will allow us to have more categories in the future. 
> This would also remove the reliance on the test class name.
> 
> Test categories skeleton:
>   SqoopTest _ UnitTest
>   |__ IntegrationTest
>   |__ ManualTest
> 
>   ThirdPartyTest _ CubridTest
>|__ Db2Test
>|__ MainFrameTest
>|__ MysqlTest
>|__ NetezzaTest
>|__ OracleTest
>|__ PostgresqlTest
>|__ SqlServerTest
> 
>   KerberizedTest
> 
> Categories explanation:
> * SqoopTest: Group of the big categories, including:
> - UnitTest: It tests one class only with its dependencies mocked or 
> if the dependency
> is lightweight we can keep it. It must not start a minicluster or an 
> hsqldb database.
> It does not need JCDB drivers.
> - IntegrationTest: It usually tests a whole scenario. It may start up 
> miniclusters,
> hsqldb and connect to external resources like RDBMSs.
> - ManualTest: This should be a deprecated category which should not 
> be used in the future.
> It only exists to mark the currently existing manual tests.
> * ThirdPartyTest: An orthogonal hierarchy for tests that need a JDBC 
> driver and/or a docker
> container/external RDBMS instance to run. Subcategories express what kind 
> of external
> resource the test needs. E.g: OracleTest needs an Oracle RDBMS and Oracle 
> driver on the classpath
> * KerberizedTest: Test that needs Kerberos, which needs to be run on a 
> separate JVM.
> 
> Opinions are very welcomed. Thanks!
> 
> 
> Diffs
> -
> 
>   build.gradle 2014b5cf5 
>   src/test/org/apache/sqoop/TestConnFactory.java fb6c94059 
>   src/test/org/apache/sqoop/TestIncrementalImport.java 29c477954 
>   src/test/org/apache/sqoop/TestSqoopOptions.java e55682edf 
>   src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java 631eeff5e 
>   src/test/org/apache/sqoop/authentication/TestKerberosAuthenticator.java 
> f5700ce65 
>   src/test/org/apache/sqoop/db/TestDriverManagerJdbcConnectionFactory.java 
> 244831672 
>   
> src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
>  d3e3fb23e 
>   src/test/org/apache/sqoop/hbase/HBaseImportAddRowKeyTest.java c4caafba5 
>   src/test/org/apache/sqoop/hbase/HBaseKerberizedConnectivityTest.java 
> 3bfb39178 
>   src/test/org/apache/sqoop/hbase/HBaseTestCase.java 94b71b61c 
>   src/test/org/apache/sqoop/hbase/HBaseUtilTest.java c6a808c33 
>   src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java e78a535f4 
>   src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java ba05cabbb 
>   
> src/test/org/apache/sqoop/hive/HiveServer2ConnectionFactoryInitializerTest.java
>  4d2cb2f88 
>   src/test/org/apache/sqoop/hive/TestHiveClientFactory.java a3c2dc939 
>   src/test/org/apache/sqoop/hive/TestHiveMiniCluster.java 419f888c0 
>   src/test/org/apache/sqoop/hive/TestHiveServer2Client.java 02617295e 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java 65f079467 
>   src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java 410724f37 
>   src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java 
> 276e9eaa4 
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java 626ad22f6 
>   src/test/org/apache/sqoop/hive/TestTableDefWriterForExternalTable.java 
> f1768ee76 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   src/test/org/apache/sqoop/io/TestCodecMap.java e71921823 
>   src/test/org/apache/sqoop/io/TestLobFile.java 2bc95f283 
>   src/test/org/apache/sqoop/io/TestNamedFifo.java a93784e08 
>   src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java c59aa26ad 
>   src/test/org/apache/sqoop/lib/TestBlobRef.java b271d3c7b 
>   

Re: Review Request 62523: SQOOP-3237: Mainframe FTP transfer option to insert custom FTP commands prior to transfer

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62523/#review210537
---



Hi Chris,

Thanks for the update, MainframeManagerImportTest passes now.

I have however a couple of new findings regarding your latest update and in 
general regarding to extra FTP commands usage, please find them below.

Thank you,
Bogi


src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java
Lines 86-90 (patched)


This block is embedded under the condition of 'if 
(SqoopOptions.FileLayout.BinaryFile == options.getFileLayout())'. Does this 
mean that these extra commands can work only with binary file? If yes, could 
you please explain why?



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 56 (patched)


You never use DEFAULT_FTP_URL anywhere.



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Line 120 (original), 129 (patched)


I think clean up of these lines has been missed. You are setting the 
username twice now. This applies for the other test cases too.



src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java
Lines 192-194 (original), 197-199 (patched)


These should have been cleaned up too I guess. This applies for the other 
test cases too.


- Boglarka Egyed


On Nov. 13, 2018, 11:24 p.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62523/
> ---
> 
> (Updated Nov. 13, 2018, 11:24 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3237
> https://issues.apache.org/jira/browse/SQOOP-3237
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Added --ftpcmds command to allow comma separated list of FTP commands to send.
> 
> 
> Diffs
> -
> 
>   src/docs/user/import-mainframe.txt 3ecfb7e4 
>   src/java/org/apache/sqoop/SqoopOptions.java f06872f9 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java 
> 9842daa6 
>   src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 
> 90dc2ddd 
>   src/java/org/apache/sqoop/tool/MainframeImportTool.java fbc8c3db 
>   src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java e7c48a6b 
>   src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 00e57bd0 
>   src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java 0714bdcf 
> 
> 
> Diff: https://reviews.apache.org/r/62523/diff/9/
> 
> 
> Testing
> ---
> 
> Unit tests.
> 
> 
> File Attachments
> 
> 
> SQOOP-3237-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/09/26/56041556-e355-4372-83ab-1bcc01680201__SQOOP-3237-1.patch
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

[jira] [Commented] (SQOOP-3382) Add parquet numeric support for Parquet in hdfs import

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686375#comment-16686375
 ] 

Hudson commented on SQOOP-3382:
---

FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1226 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1226/])
SQOOP-3382: Add parquet numeric support for Parquet in hdfs import (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=00a02dec2f7507f813ee4899096c470ba1112a9e])
* (edit) src/test/org/apache/sqoop/util/ParquetReader.java
* (edit) src/java/org/apache/sqoop/config/ConfigurationConstants.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
* (edit) src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
* (edit) 
src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java
* (delete) src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/OracleImportJobTestConfigurationForNumber.java
* (add) src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java
* (edit) src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/ImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/OracleImportJobTestConfiguration.java
* (edit) src/test/org/apache/sqoop/importjob/SplitByImportTest.java
* (edit) 
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
* (delete) 
src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java


> Add parquet numeric support for Parquet in hdfs import
> --
>
> Key: SQOOP-3382
> URL: https://issues.apache.org/jira/browse/SQOOP-3382
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> The current Avro numeric tests are suitable to be used as Parquet tests, with 
> very minor modifications, as parquet can be written with the same input and 
> nearly the same args. Since we are writing Parquet with it's Avro support, it 
> would be good to cover this code with the same, or similar tests (including 
> the edge cases related to padding, missing scale and precision cases).
> Differences are:
>  * the expected output, since stored in a parquet file is different.
>  * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/#review210533
---


Ship it!




Thanks for the updates Fero, let's ship this!

- Boglarka Egyed


On Nov. 12, 2018, 4:33 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69060/
> ---
> 
> (Updated Nov. 12, 2018, 4:33 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3382
> https://issues.apache.org/jira/browse/SQOOP-3382
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch is about adding support for fixed point decimal types in parquet 
> import.
> 
> The implementation is simple after the fact that parquet was upgraded to 
> 1.9.0 in SQOOP-3381: we just need to register the GenericDataSupplier with 
> AvroParquetOutputFormat.
> 
> For testing, we can reuse the existing Avro tests, because Sqoop uses Avro 
> under the hood to write parquet.
> 
> I also moved around and renamed the classes involved in this change so their 
> name and package reflect their purpose.
> 
> ** Note: A key design decision can be seen in the ImportJobTestConfiguration 
> interface **
> - I decided to create a new function to get the expected results for each 
> file format, since we seldom add new fileformats. 
> - However this also enforces future configurations to always define their 
> expected result for every file forma or throw a NotImplementedException 
> should they lack the support for one.
> - The alternative for this is to define the fileLayout as an input parameter 
> instead. This would allow for better extendability.
> _Please share your thoughts on this!_
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250e 
>   src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8ab 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
>  e82154309 
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9cd 
>   src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 
> 14de910b9 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 7977c0b0f 
>   src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java 
> ff13dc3bc 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
>  182d2967f 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
>  e9bf9912a 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
>  b7bad08c0 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
>  465e61f4b 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
>  66715c171 
>   
> src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
>  ec4db41bd 
>   
> src/test/org/apache/sqoop/importjob/configuration/AvroTestConfiguration.java 
> PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/GenericImportJobSplitByTestConfiguration.java
>  f137b56b7 
>   
> src/test/org/apache/sqoop/importjob/configuration/ParquetTestConfiguration.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/util/ParquetReader.java 908ce566f 
> 
> 
> Diff: https://reviews.apache.org/r/69060/diff/4/
> 
> 
> Testing
> ---
> 
> 3rd party tests and unit tests, both gradle and ant
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

[jira] [Commented] (SQOOP-3382) Add parquet numeric support for Parquet in hdfs import

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686325#comment-16686325
 ] 

ASF subversion and git services commented on SQOOP-3382:


Commit 00a02dec2f7507f813ee4899096c470ba1112a9e in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=00a02de ]

SQOOP-3382: Add parquet numeric support for Parquet in hdfs import

(Fero Szabo via Szabolcs Vasas)


> Add parquet numeric support for Parquet in hdfs import
> --
>
> Key: SQOOP-3382
> URL: https://issues.apache.org/jira/browse/SQOOP-3382
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> The current Avro numeric tests are suitable to be used as Parquet tests, with 
> very minor modifications, as parquet can be written with the same input and 
> nearly the same args. Since we are writing Parquet with it's Avro support, it 
> would be good to cover this code with the same, or similar tests (including 
> the edge cases related to padding, missing scale and precision cases).
> Differences are:
>  * the expected output, since stored in a parquet file is different.
>  * input arguements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)