Review Request 68541: SQOOP-3104: Create test categories instead of test suites and naming conventions

2018-08-28 Thread Nguyen Truong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68541/
---

Review request for Sqoop.


Bugs: SQOOP-3104
https://issues.apache.org/jira/browse/SQOOP-3104


Repository: sqoop-trunk


Description
---

We are currently unsing test naming conventions to differentiate between 
ManualTests, Unit tests and 3rd party tests. Instead of that, I implemented 
junit categories which will allow us to have more categories in the future. 
This would also remove the reliance on the test class name.

Test categories skeleton:
  SqoopTest _ UnitTest
  |__ IntegrationTest
  |__ ManualTest

  ThirdPartyTest _ CubridTest
   |__ Db2Test
   |__ MainFrameTest
   |__ MysqlTest
   |__ NetezzaTest
   |__ OracleTest
   |__ PostgresqlTest
   |__ SqlServerTest

  KerberizedTest

Categories explanation:
* SqoopTest: Group of the big categories, including:
- UnitTest: It tests one class only with its dependencies mocked or if 
the dependency
is lightweight we can keep it. It must not start a minicluster or an 
hsqldb database.
It does not need JCDB drivers.
- IntegrationTest: It usually tests a whole scenario. It may start up 
miniclusters,
hsqldb and connect to external resources like RDBMSs.
- ManualTest: This should be a deprecated category which should not be 
used in the future.
It only exists to mark the currently existing manual tests.
* ThirdPartyTest: An orthogonal hierarchy for tests that need a JDBC driver 
and/or a docker
container/external RDBMS instance to run. Subcategories express what kind 
of external
resource the test needs. E.g: OracleTest needs an Oracle RDBMS and Oracle 
driver on the classpath
* KerberizedTest: Test that needs Kerberos, which needs to be run on a 
separate JVM.

Opinions are very welcomed. Thanks!


Diffs
-

  build.gradle fc7fc0c4c 
  src/test/org/apache/sqoop/TestConnFactory.java fb6c94059 
  src/test/org/apache/sqoop/TestIncrementalImport.java 29c477954 
  src/test/org/apache/sqoop/TestSqoopOptions.java e55682edf 
  src/test/org/apache/sqoop/accumulo/TestAccumuloUtil.java 631eeff5e 
  src/test/org/apache/sqoop/authentication/TestKerberosAuthenticator.java 
f5700ce65 
  src/test/org/apache/sqoop/db/TestDriverManagerJdbcConnectionFactory.java 
244831672 
  
src/test/org/apache/sqoop/db/decorator/TestKerberizedConnectionFactoryDecorator.java
 d3e3fb23e 
  src/test/org/apache/sqoop/hbase/HBaseKerberizedConnectivityTest.java 
3bfb39178 
  src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java e78a535f4 
  src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java ba05cabbb 
  
src/test/org/apache/sqoop/hive/HiveServer2ConnectionFactoryInitializerTest.java 
4d2cb2f88 
  src/test/org/apache/sqoop/hive/TestHiveClientFactory.java a3c2dc939 
  src/test/org/apache/sqoop/hive/TestHiveMiniCluster.java 419f888c0 
  src/test/org/apache/sqoop/hive/TestHiveServer2Client.java 02617295e 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4f 
  src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java 410724f37 
  src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java 276e9eaa4 
  src/test/org/apache/sqoop/hive/TestTableDefWriter.java 626ad22f6 
  src/test/org/apache/sqoop/hive/TestTableDefWriterForExternalTable.java 
f1768ee76 
  src/test/org/apache/sqoop/io/TestCodecMap.java e71921823 
  src/test/org/apache/sqoop/io/TestLobFile.java 2bc95f283 
  src/test/org/apache/sqoop/io/TestNamedFifo.java a93784e08 
  src/test/org/apache/sqoop/io/TestSplittableBufferedWriter.java c59aa26ad 
  src/test/org/apache/sqoop/lib/TestBlobRef.java b271d3c7b 
  src/test/org/apache/sqoop/lib/TestBooleanParser.java 914ab37e4 
  src/test/org/apache/sqoop/lib/TestClobRef.java f94d1a8af 
  src/test/org/apache/sqoop/lib/TestFieldFormatter.java 9ac55e703 
  src/test/org/apache/sqoop/lib/TestLargeObjectLoader.java 1e07d7174 
  src/test/org/apache/sqoop/lib/TestRecordParser.java d6844c1cf 
  src/test/org/apache/sqoop/manager/TestDefaultManagerFactory.java 8e1632430 
  src/test/org/apache/sqoop/manager/TestMainframeManager.java c84f05f66 
  src/test/org/apache/sqoop/manager/TestSqlManager.java 185f5a7a1 
  src/test/org/apache/sqoop/manager/cubrid/CubridAuthTest.java 82fac12e3 
  src/test/org/apache/sqoop/manager/cubrid/CubridCompatTest.java 8a075e87d 
  src/test/org/apache/sqoop/manager/cubrid/CubridManagerExportTest.java 
4de8e40fd 
  src/test/org/apache/sqoop/manager/cubrid/CubridManagerImportTest.java 
addf1aeec 
  
src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
 d1a6d6926

Re: Review Request 68536: SQOOP-3368: Add fail-fast scenarios to S3 incremental import use cases without --temporary-rootdir option

2018-08-28 Thread Szabolcs Vasas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68536/#review208043
---



Hi Bogi,

Thanks for submitting this patch, it fills an important gap in Sqoop's S3 
support.
Please see my below findings:


src/java/org/apache/sqoop/tool/ImportTool.java
Lines 1170 (patched)


Can we somehow simplify and merge these 2 if statements?
Ideas:
- since IncrementalMode has only 3 possible values checking if 
options.getIncrementalMode() != SqoopOptions.IncrementalMode.None might be 
enough
- by using org.apache.commons.lang3.StringUtils#contains we could avoid 
null checks



src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java
Lines 97 (patched)


Just for the sake of completeness we could add test case(s) to verify if no 
exception is thrown when everything is OK.


- Szabolcs Vasas


On Aug. 28, 2018, 11:56 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68536/
> ---
> 
> (Updated Aug. 28, 2018, 11:56 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3368
> https://issues.apache.org/jira/browse/SQOOP-3368
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The current implementation of Sqoop handles HDFS as a default filesystem, 
> i.e. it creates temporary directories on HDFS in case of incremental append 
> or merge imports. To make these incremental import use cases work with S3 the 
> user needs to set the --temporary-rootdir to an S3 location properly.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/tool/ImportTool.java 
> 139733732d2a28d171568b9118c98a47a3d2fc50 
>   
> src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68536/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test
> ./gradlew test
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Resolved] (SQOOP-3365) Improve Sqoop tests to be able to execute them in a single JVM

2018-08-28 Thread Szabolcs Vasas (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SQOOP-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3365.
---
Resolution: Fixed

> Improve Sqoop tests to be able to execute them in a single JVM
> --
>
> Key: SQOOP-3365
> URL: https://issues.apache.org/jira/browse/SQOOP-3365
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Priority: Major
>
> Gradle currently creates a new JVM for every test case it executes (the 
> forkEvery parameter is set to 1, see: 
> https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery).
>  This provides better isolation for the tests but it has huge performance 
> overhead as well since creating a JVM is an expensive operation. It would be 
> great if we could execute all our tests in a single JVM so we could save that 
> big cost but unfortunately it does not work out of the box because:
>  * There are test classes which unnecessarily change the state of the JVM 
> (for example by setting static fields and system properties) and can cause 
> other tests to fail if they are run in the same JVM.
>  * There are test classes which rely on a "clean JVM state" and they fail if 
> the state is different than implicitly expected by them.
>  * There are test classes which create a Kerberos KDC which sets a lot of 
> static fields in the JVM which are really hard if not impossible to restore.
> This JIRA addresses the first two of the above issues.
> I expect the third category to be addressed by SQOOP-3104. It should 
> introduce a category for kerberized tests and Gradle should execute this 
> category with the forkEvery parameter set to 1.
> However Gradle should be able to execute the rest of the tests with forkEvery 
> 0.
> Please note that the test execution order with Gradle is not strictly defined 
> but platform dependent so it is possible that running the tests on a new 
> platform will produce an execution order which will unveil a test 
> interdependency which is not resolved by the subtasks of the JIRA.
> If you encounter such a situation please feel free to create a new subtasks 
> here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

2018-08-28 Thread Shashank (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595029#comment-16595029
 ] 

Shashank commented on SQOOP-2331:
-

[~fero] Yes , I am still working on it.The problem in adding compression test 
cases in class HCatalogImportTest is that it returns HCatRecord. The HCatRecord 
class does not have any compression or file properties , so I am still figure 
out ways how to get compression type from the output. Any suggestions will be 
helpful.

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.7
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch, SQOOP-2331_2.patch, SQOOP-2331_3.patch
>
>
> Current Apache Sqoop 1.4.7 does not compress in gzip format with 
>  --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3367) Improve third party tests to be able to execute them in a single JVM

2018-08-28 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595000#comment-16595000
 ] 

Hudson commented on SQOOP-3367:
---

SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1203 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1203/])
SQOOP-3367: Improve third party tests to be able to execute them in a (bogi: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=327aec8bf1b7a40936503a9c5a94b759caf16e11])
* (edit) testdata/hive/scripts/normalImport.q
* (edit) testdata/hive/scripts/numericImport.q
* (edit) testdata/hive/scripts/customDelimImport.q
* (edit) testdata/hive/scripts/failingImport.q
* (edit) src/java/org/apache/sqoop/hive/HiveImport.java
* (edit) src/test/org/apache/sqoop/manager/oracle/SystemImportTest.java
* (edit) testdata/hive/scripts/partitionImport.q
* (edit) testdata/hive/scripts/fieldWithNewlineImport.q
* (edit) testdata/hive/scripts/incrementalHiveAppend20.q
* (edit) src/test/org/apache/sqoop/testutil/LobAvroImportTestCase.java
* (edit) testdata/hive/scripts/dateImport.q
* (edit) testdata/hive/scripts/fieldWithNewlineReplacementImport.q
* (edit) src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java
* (edit) src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java
* (edit) testdata/hive/scripts/incrementalHiveAppendEmpty.q
* (edit) testdata/hive/scripts/incrementalHiveAppend10.q
* (edit) testdata/hive/scripts/decimalMapImport.q


> Improve third party tests to be able to execute them in a single JVM
> 
>
> Key: SQOOP-3367
> URL: https://issues.apache.org/jira/browse/SQOOP-3367
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3367.patch
>
>
> The goal of this JIRA is to improve the third party tests to be able to 
> execute them in a single JVM. See the parent JIRA for the details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Amit Joshi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594982#comment-16594982
 ] 

Amit Joshi commented on SQOOP-3042:
---

[~dvoros] Thanks but getting error while assigning hdfs dir path to the 
--bindir option.

Its java.lang.reflect.InvocationTargetException

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-2331) Snappy Compression Support in Sqoop-HCatalog

2018-08-28 Thread Fero Szabo (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594979#comment-16594979
 ] 

Fero Szabo commented on SQOOP-2331:
---

Hi [~standon],

I've reviewed the modified code and made a few suggestions on ReviewBoard. Can 
you please have a look? 

Also, I wonder if you saw the update there when I posted it, as it was two 
weeks ago... 

Thanks,

Fero

> Snappy Compression Support in Sqoop-HCatalog
> 
>
> Key: SQOOP-2331
> URL: https://issues.apache.org/jira/browse/SQOOP-2331
> Project: Sqoop
>  Issue Type: New Feature
>Affects Versions: 1.4.7
>Reporter: Atul Gupta
>Assignee: Shashank
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-2331_0.patch, SQOOP-2331_1.patch, 
> SQOOP-2331_2.patch, SQOOP-2331_2.patch, SQOOP-2331_3.patch
>
>
> Current Apache Sqoop 1.4.7 does not compress in gzip format with 
>  --compress option while using with --hcatalog-table option. It also does not 
> support option --compression-codec snappy with --hcatalog-table option. it 
> would be nice if we add both the options in the Sqoop future releases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3367) Improve third party tests to be able to execute them in a single JVM

2018-08-28 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594974#comment-16594974
 ] 

ASF subversion and git services commented on SQOOP-3367:


Commit 327aec8bf1b7a40936503a9c5a94b759caf16e11 in sqoop's branch 
refs/heads/trunk from [~BoglarkaEgyed]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=327aec8 ]

SQOOP-3367: Improve third party tests to be able to execute them in a single JVM

(Szabolcs Vasas via Boglarka Egyed)


> Improve third party tests to be able to execute them in a single JVM
> 
>
> Key: SQOOP-3367
> URL: https://issues.apache.org/jira/browse/SQOOP-3367
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3367.patch
>
>
> The goal of this JIRA is to improve the third party tests to be able to 
> execute them in a single JVM. See the parent JIRA for the details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3363) Test incremental import with S3

2018-08-28 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594969#comment-16594969
 ] 

Hudson commented on SQOOP-3363:
---

SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1202 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1202/])
SQOOP-3363: Test incremental import with S3 (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=816146df567cf46a972d16873c46595f778b136c])
* (edit) src/test/org/apache/sqoop/TestAppendUtils.java
* (edit) src/test/org/apache/sqoop/testutil/TextFileTestUtils.java
* (add) src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java
* (edit) src/test/org/apache/sqoop/testutil/AvroTestUtils.java
* (add) src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java
* (edit) src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java
* (edit) src/java/org/apache/sqoop/util/FileSystemUtil.java
* (edit) src/test/org/apache/sqoop/s3/TestS3TextImport.java
* (edit) src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java
* (edit) src/java/org/apache/sqoop/util/AppendUtils.java
* (add) src/test/org/apache/sqoop/s3/TestS3ParquetImport.java
* (add) src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java
* (edit) src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java
* (add) src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java
* (add) 
src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java
* (edit) src/test/org/apache/sqoop/testutil/S3TestUtils.java
* (edit) src/test/org/apache/sqoop/s3/TestS3AvroImport.java
* (add) src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java


> Test incremental import with S3
> ---
>
> Key: SQOOP-3363
> URL: https://issues.apache.org/jira/browse/SQOOP-3363
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: SQOOP-3363.patch, SQOOP-3363.patch, SQOOP-3363.patch, 
> SQOOP-3363.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3373) Document simple and incremental import into S3

2018-08-28 Thread Boglarka Egyed (JIRA) 
Boglarka Egyed created SQOOP-3373:
-

 Summary: Document simple and incremental import into S3
 Key: SQOOP-3373
 URL: https://issues.apache.org/jira/browse/SQOOP-3373
 Project: Sqoop
  Issue Type: Sub-task
Affects Versions: 1.4.7
Reporter: Boglarka Egyed
Assignee: Boglarka Egyed






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3363) Test incremental import with S3

2018-08-28 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594899#comment-16594899
 ] 

ASF subversion and git services commented on SQOOP-3363:


Commit 816146df567cf46a972d16873c46595f778b136c in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=816146d ]

SQOOP-3363: Test incremental import with S3

(Boglarka Egyed via Szabolcs Vasas)


> Test incremental import with S3
> ---
>
> Key: SQOOP-3363
> URL: https://issues.apache.org/jira/browse/SQOOP-3363
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3363.patch, SQOOP-3363.patch, SQOOP-3363.patch, 
> SQOOP-3363.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3363) Test incremental import with S3

2018-08-28 Thread Boglarka Egyed (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SQOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3363:
--
Attachment: SQOOP-3363.patch

> Test incremental import with S3
> ---
>
> Key: SQOOP-3363
> URL: https://issues.apache.org/jira/browse/SQOOP-3363
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3363.patch, SQOOP-3363.patch, SQOOP-3363.patch, 
> SQOOP-3363.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

2018-08-28 Thread Boglarka Egyed 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/
---

(Updated Aug. 28, 2018, 12:04 p.m.)


Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and Szabolcs 
Vasas.


Changes
---

Indentation fixed in S3TestUtils


Bugs: SQOOP-3363
https://issues.apache.org/jira/browse/SQOOP-3363


Repository: sqoop-trunk


Description
---

* Added new test cases for Parquet import into S3 as it was still missing
* Added new test cases for incremental append import into S3 in Text, Avro, 
Sequence and Parquet file format
* Added new test cases for incremental merge import into S3 in Text and Parquet 
file format
* Updated some previously added logic in S3 util and test classes


Diffs (updated)
-

  src/java/org/apache/sqoop/util/AppendUtils.java 
20c0d13c391d312a943a147882cf85f86cb7e438 
  src/java/org/apache/sqoop/util/FileSystemUtil.java 
96ec21233d6378865195008f5ab67a74370fa2ed 
  src/test/org/apache/sqoop/TestAppendUtils.java 
3d66beca0226dd9eb8a14f4157444f24a247491a 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
e130c42104b86e854d45babc009a5f1409a74a48 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
c17c1c54918df0b4d1ecbaef4e381975d72756ae 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
  src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
  src/test/org/apache/sqoop/testutil/S3TestUtils.java 
ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
  src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
ad7576dbb2447423c677429f24163031a9d39b5f 
  src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 


Diff: https://reviews.apache.org/r/68475/diff/6/

Changes: https://reviews.apache.org/r/68475/diff/5-6/


Testing
---

ant clean test -Ds3.bucket.url= 
-Ds3.generator.command=
./gradlew test -Ds3.bucket.url= 
-Ds3.generator.command=


Thanks,

Boglarka Egyed

[jira] [Updated] (SQOOP-3368) Add fail-fast scenarios to S3 incremental import use cases without --temporary-rootdir option

2018-08-28 Thread Boglarka Egyed (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SQOOP-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3368:
--
Description: 
The current implementation of Sqoop handles HDFS as a default filesystem, i.e. 
it creates temporary directories on HDFS in case of incremental append or merge 
imports. To make these incremental import use cases work with S3 the user needs 
to set the {{--temporary-rootdir}} to an S3 location properly.

There should be fail-fast scenarios without the {{--temporary-rootdir}} option.

  was:
The current implementation of Sqoop handles HDFS as a default filesystem, i.e. 
it creates temporary directories on HDFS in case of incremental append or merge 
imports. To make these incremental import use cases work with S3 the user needs 
to set the {{--temporary-rootdir}} to an S3 location properly.

There should be fail-fast scenarios without the \{{--temporary-rootdir}} option 
as well as a documentation of this.


> Add fail-fast scenarios to S3 incremental import use cases without 
> --temporary-rootdir option
> -
>
> Key: SQOOP-3368
> URL: https://issues.apache.org/jira/browse/SQOOP-3368
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3368.patch
>
>
> The current implementation of Sqoop handles HDFS as a default filesystem, 
> i.e. it creates temporary directories on HDFS in case of incremental append 
> or merge imports. To make these incremental import use cases work with S3 the 
> user needs to set the {{--temporary-rootdir}} to an S3 location properly.
> There should be fail-fast scenarios without the {{--temporary-rootdir}} 
> option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3368) Add fail-fast scenarios to S3 incremental import use cases without --temporary-rootdir option

2018-08-28 Thread Boglarka Egyed (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SQOOP-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3368:
--
Attachment: SQOOP-3368.patch

> Add fail-fast scenarios to S3 incremental import use cases without 
> --temporary-rootdir option
> -
>
> Key: SQOOP-3368
> URL: https://issues.apache.org/jira/browse/SQOOP-3368
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3368.patch
>
>
> The current implementation of Sqoop handles HDFS as a default filesystem, 
> i.e. it creates temporary directories on HDFS in case of incremental append 
> or merge imports. To make these incremental import use cases work with S3 the 
> user needs to set the {{--temporary-rootdir}} to an S3 location properly.
> There should be fail-fast scenarios without the {{--temporary-rootdir}} 
> option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68536: SQOOP-3368: Add fail-fast scenarios to S3 incremental import use cases without --temporary-rootdir option

2018-08-28 Thread Boglarka Egyed 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68536/
---

Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and Szabolcs 
Vasas.


Bugs: SQOOP-3368
https://issues.apache.org/jira/browse/SQOOP-3368


Repository: sqoop-trunk


Description
---

The current implementation of Sqoop handles HDFS as a default filesystem, i.e. 
it creates temporary directories on HDFS in case of incremental append or merge 
imports. To make these incremental import use cases work with S3 the user needs 
to set the --temporary-rootdir to an S3 location properly.


Diffs
-

  src/java/org/apache/sqoop/tool/ImportTool.java 
139733732d2a28d171568b9118c98a47a3d2fc50 
  src/test/org/apache/sqoop/tool/TestS3IncrementalImportOptionValidations.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68536/diff/1/


Testing
---

ant clean test
./gradlew test


Thanks,

Boglarka Egyed

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

2018-08-28 Thread Fero Szabo via Review Board 


> On Aug. 28, 2018, 11:41 a.m., Fero Szabo wrote:
> >

All ant tests passed on my system. (unit, 3rd party and s3). Gradle unit tests 
passed as well.


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/#review208030
---


On Aug. 28, 2018, 8:33 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68475/
> ---
> 
> (Updated Aug. 28, 2018, 8:33 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3363
> https://issues.apache.org/jira/browse/SQOOP-3363
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> * Added new test cases for Parquet import into S3 as it was still missing
> * Added new test cases for incremental append import into S3 in Text, Avro, 
> Sequence and Parquet file format
> * Added new test cases for incremental merge import into S3 in Text and 
> Parquet file format
> * Updated some previously added logic in S3 util and test classes
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java 
> 20c0d13c391d312a943a147882cf85f86cb7e438 
>   src/java/org/apache/sqoop/util/FileSystemUtil.java 
> 96ec21233d6378865195008f5ab67a74370fa2ed 
>   src/test/org/apache/sqoop/TestAppendUtils.java 
> 3d66beca0226dd9eb8a14f4157444f24a247491a 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> e130c42104b86e854d45babc009a5f1409a74a48 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> c17c1c54918df0b4d1ecbaef4e381975d72756ae 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
>   src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
> 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
> ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
>   src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
> ad7576dbb2447423c677429f24163031a9d39b5f 
>   src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
> df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 
> 
> 
> Diff: https://reviews.apache.org/r/68475/diff/5/
> 
> 
> Testing
> ---
> 
> ant clean test -Ds3.bucket.url= 
> -Ds3.generator.command=
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

2018-08-28 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/#review208030
---


Fix it, then Ship it!





src/test/org/apache/sqoop/testutil/S3TestUtils.java
Lines 181-185 (patched)


nit: indentation missing.


- Fero Szabo


On Aug. 28, 2018, 8:33 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68475/
> ---
> 
> (Updated Aug. 28, 2018, 8:33 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3363
> https://issues.apache.org/jira/browse/SQOOP-3363
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> * Added new test cases for Parquet import into S3 as it was still missing
> * Added new test cases for incremental append import into S3 in Text, Avro, 
> Sequence and Parquet file format
> * Added new test cases for incremental merge import into S3 in Text and 
> Parquet file format
> * Updated some previously added logic in S3 util and test classes
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java 
> 20c0d13c391d312a943a147882cf85f86cb7e438 
>   src/java/org/apache/sqoop/util/FileSystemUtil.java 
> 96ec21233d6378865195008f5ab67a74370fa2ed 
>   src/test/org/apache/sqoop/TestAppendUtils.java 
> 3d66beca0226dd9eb8a14f4157444f24a247491a 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> e130c42104b86e854d45babc009a5f1409a74a48 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> c17c1c54918df0b4d1ecbaef4e381975d72756ae 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
>   src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
> 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
> ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
>   src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
> ad7576dbb2447423c677429f24163031a9d39b5f 
>   src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
> df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 
> 
> 
> Diff: https://reviews.apache.org/r/68475/diff/5/
> 
> 
> Testing
> ---
> 
> ant clean test -Ds3.bucket.url= 
> -Ds3.generator.command=
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

2018-08-28 Thread Szabolcs Vasas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/#review208029
---


Ship it!




Hi Bogi,

Thanks for fixing the findings, I have reran the tests again, all green.

- Szabolcs Vasas


On Aug. 28, 2018, 8:33 a.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68475/
> ---
> 
> (Updated Aug. 28, 2018, 8:33 a.m.)
> 
> 
> Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3363
> https://issues.apache.org/jira/browse/SQOOP-3363
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> * Added new test cases for Parquet import into S3 as it was still missing
> * Added new test cases for incremental append import into S3 in Text, Avro, 
> Sequence and Parquet file format
> * Added new test cases for incremental merge import into S3 in Text and 
> Parquet file format
> * Updated some previously added logic in S3 util and test classes
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java 
> 20c0d13c391d312a943a147882cf85f86cb7e438 
>   src/java/org/apache/sqoop/util/FileSystemUtil.java 
> 96ec21233d6378865195008f5ab67a74370fa2ed 
>   src/test/org/apache/sqoop/TestAppendUtils.java 
> 3d66beca0226dd9eb8a14f4157444f24a247491a 
>   src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
> e130c42104b86e854d45babc009a5f1409a74a48 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
> c17c1c54918df0b4d1ecbaef4e381975d72756ae 
>   src/test/org/apache/sqoop/s3/TestS3TextImport.java 
> 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
>   src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
> 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
> ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
>   src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
> ad7576dbb2447423c677429f24163031a9d39b5f 
>   src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
> df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 
> 
> 
> Diff: https://reviews.apache.org/r/68475/diff/5/
> 
> 
> Testing
> ---
> 
> ant clean test -Ds3.bucket.url= 
> -Ds3.generator.command=
> ./gradlew test -Ds3.bucket.url= 
> -Ds3.generator.command=
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Daniel Voros (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594827#comment-16594827
 ] 

Daniel Voros commented on SQOOP-3042:
-

[~amjosh911] use the `--bindir` option, see 
[here|https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html].

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Amit Joshi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594785#comment-16594785
 ] 

Amit Joshi commented on SQOOP-3042:
---

Is there a sqoop property available to configure the compile directory?

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Amit Joshi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594781#comment-16594781
 ] 

Amit Joshi commented on SQOOP-3042:
---

[~dvoros] Thanks for this info

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68516: Improve third party tests to be able to execute them in a single JVM

2018-08-28 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68516/#review208019
---


Ship it!




Unit and S3 tests passed for me as well. (Except for the ones with Kerberos, of 
course.)

- Fero Szabo


On Aug. 27, 2018, 12:38 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68516/
> ---
> 
> (Updated Aug. 27, 2018, 12:38 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3367
> https://issues.apache.org/jira/browse/SQOOP-3367
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The goal of this JIRA is to improve the third party tests to be able to 
> execute them in a single JVM. See the parent JIRA for the details.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/hive/HiveImport.java 5da00a740 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java a8072b92a 
>   src/test/org/apache/sqoop/manager/oracle/SystemImportTest.java e0a0462c3 
>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java ad2f10a07 
>   src/test/org/apache/sqoop/testutil/LobAvroImportTestCase.java 20d6151ba 
>   testdata/hive/scripts/customDelimImport.q a27396947 
>   testdata/hive/scripts/dateImport.q 476d4310c 
>   testdata/hive/scripts/decimalMapImport.q 8f05d585d 
>   testdata/hive/scripts/failingImport.q 476d4310c 
>   testdata/hive/scripts/fieldWithNewlineImport.q c6c9ebdaa 
>   testdata/hive/scripts/fieldWithNewlineReplacementImport.q a4b8a5992 
>   testdata/hive/scripts/incrementalHiveAppend10.q 383c7b4aa 
>   testdata/hive/scripts/incrementalHiveAppend20.q 383c7b4aa 
>   testdata/hive/scripts/incrementalHiveAppendEmpty.q 383c7b4aa 
>   testdata/hive/scripts/normalImport.q 34d6ac1ea 
>   testdata/hive/scripts/numericImport.q bef7c715e 
>   testdata/hive/scripts/partitionImport.q 68ce7c1bb 
> 
> 
> Diff: https://reviews.apache.org/r/68516/diff/1/
> 
> 
> Testing
> ---
> 
> Ran unit and third party tests.
> Note that if you want to run the tests with forkEvery set to 0, you have to 
> disable the kereberized test cases first as I did here: 
> https://github.com/szvasas/sqoop/commit/d9ba8a511ed0bad119e105b97f4725ea0951
> Setting the forkEvery to 0: 
> https://github.com/szvasas/sqoop/commit/6332c2959bafb95e83c2d9a4030ba432cfd4c640
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Daniel Voros (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594718#comment-16594718
 ] 

Daniel Voros commented on SQOOP-3042:
-

[~amjosh911] it is going to be included in the next release we do from trunk. 
Not sure yet if it's going to be 1.4.8, 1.5.0 or 3.0.0.

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-08-28 Thread Amit Joshi (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594714#comment-16594714
 ] 

Amit Joshi commented on SQOOP-3042:
---

Hi guys, may I know which version of sqoop has this change incorporated into?

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68475: SQOOP-3363: Test incremental import with S3

2018-08-28 Thread Boglarka Egyed 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68475/
---

(Updated Aug. 28, 2018, 8:33 a.m.)


Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and Szabolcs 
Vasas.


Changes
---

Bigger refactor based on latest reviews: simplified test case responsibilities, 
eliminated code duplications, rationalized constant usage.


Bugs: SQOOP-3363
https://issues.apache.org/jira/browse/SQOOP-3363


Repository: sqoop-trunk


Description
---

* Added new test cases for Parquet import into S3 as it was still missing
* Added new test cases for incremental append import into S3 in Text, Avro, 
Sequence and Parquet file format
* Added new test cases for incremental merge import into S3 in Text and Parquet 
file format
* Updated some previously added logic in S3 util and test classes


Diffs (updated)
-

  src/java/org/apache/sqoop/util/AppendUtils.java 
20c0d13c391d312a943a147882cf85f86cb7e438 
  src/java/org/apache/sqoop/util/FileSystemUtil.java 
96ec21233d6378865195008f5ab67a74370fa2ed 
  src/test/org/apache/sqoop/TestAppendUtils.java 
3d66beca0226dd9eb8a14f4157444f24a247491a 
  src/test/org/apache/sqoop/s3/TestS3AvroImport.java 
e130c42104b86e854d45babc009a5f1409a74a48 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java 
PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION 
  src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java 
c17c1c54918df0b4d1ecbaef4e381975d72756ae 
  src/test/org/apache/sqoop/s3/TestS3TextImport.java 
60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 
  src/test/org/apache/sqoop/testutil/AvroTestUtils.java 
04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 
ad2f10a071a0859c5b5e063c8cf5dda7c202124f 
  src/test/org/apache/sqoop/testutil/S3TestUtils.java 
ceaff3b3a2bfd031b9772c9b43afdfa670c23718 
  src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java 
ad7576dbb2447423c677429f24163031a9d39b5f 
  src/test/org/apache/sqoop/testutil/TextFileTestUtils.java 
df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a 


Diff: https://reviews.apache.org/r/68475/diff/5/

Changes: https://reviews.apache.org/r/68475/diff/4-5/


Testing
---

ant clean test -Ds3.bucket.url= 
-Ds3.generator.command=
./gradlew test -Ds3.bucket.url= 
-Ds3.generator.command=


Thanks,

Boglarka Egyed