Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename

2018-07-12 Thread Eric Lin via Review Board 


> On July 10, 2018, 4:05 p.m., Boglarka Egyed wrote:
> > .gitignore
> > Lines 39 (patched)
> > 
> >
> > Just as in https://reviews.apache.org/r/54528/ why do we need this here?
> 
> Eric Lin wrote:
> It happened to me a few times that "out" directory was created after 
> running "ant jar-all" or "ant test" commands, as those are the ones I run 
> most often. However, not every time though, so I am not sure what exactly 
> will create this directory, but definitely happened to me several times. (I 
> tested again, it was not created anymore)

Removed


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/#review205902
---


On July 13, 2018, 5:25 a.m., Eric Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67424/
> ---
> 
> (Updated July 13, 2018, 5:25 a.m.)
> 
> 
> Review request for Sqoop and Boglarka Egyed.
> 
> 
> Bugs: SQOOP-3330
> https://issues.apache.org/jira/browse/SQOOP-3330
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> When adding --append to Sqoop directory import with 
> -Dmapreduce.output.basename, all files will be ignored, which end up nothing 
> being imported. See below DEBUG output:
> 
> {code}
> sqoop import -Dmapreduce.output.basename="eric-test" --connect 
> jdbc:mysql://mysql-host.com/test --username root --password 'root' --table 
> test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' 
> --verbose --append
> 
> 18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-0 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-1 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-2 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 
> 14935e396acc4ea7b9a6236c66064c9b_test
> {code}
> 
> This is due to AppendUtils only recognizes file name starts with 
> "part.*-([0-9]"
> 
> https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
>   src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 
> 
> 
> Diff: https://reviews.apache.org/r/67424/diff/4/
> 
> 
> Testing
> ---
> 
> Added test case + manual test in VM
> 
> 
> Thanks,
> 
> Eric Lin
> 
>

Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename

2018-07-12 Thread Eric Lin via Review Board 


> On July 12, 2018, 3:41 p.m., Szabolcs Vasas wrote:
> > src/test/org/apache/sqoop/TestAppendUtils.java
> > Lines 331 (patched)
> > 
> >
> > Nit: can you please use generic List here, otherwise the compiler gives 
> > a warning:
> > List args = new ArrayList<>();

I updated to ArrayList args = new ArrayList<>(); instead of 
List args = new ArrayList<>();, since List will cause error in 
function runAppendTest(args, output, prefix).


- Eric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/#review206012
---


On July 13, 2018, 5:25 a.m., Eric Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67424/
> ---
> 
> (Updated July 13, 2018, 5:25 a.m.)
> 
> 
> Review request for Sqoop and Boglarka Egyed.
> 
> 
> Bugs: SQOOP-3330
> https://issues.apache.org/jira/browse/SQOOP-3330
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> When adding --append to Sqoop directory import with 
> -Dmapreduce.output.basename, all files will be ignored, which end up nothing 
> being imported. See below DEBUG output:
> 
> {code}
> sqoop import -Dmapreduce.output.basename="eric-test" --connect 
> jdbc:mysql://mysql-host.com/test --username root --password 'root' --table 
> test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' 
> --verbose --append
> 
> 18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-0 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-1 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-2 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 
> 14935e396acc4ea7b9a6236c66064c9b_test
> {code}
> 
> This is due to AppendUtils only recognizes file name starts with 
> "part.*-([0-9]"
> 
> https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
>   src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 
> 
> 
> Diff: https://reviews.apache.org/r/67424/diff/4/
> 
> 
> Testing
> ---
> 
> Added test case + manual test in VM
> 
> 
> Thanks,
> 
> Eric Lin
> 
>

Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename

2018-07-12 Thread Eric Lin via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/
---

(Updated July 13, 2018, 5:25 a.m.)


Review request for Sqoop and Boglarka Egyed.


Changes
---

update from List args = new ArrayList<>(); to ArrayList args = 
new ArrayList<>() to avoid compiler warning.


Bugs: SQOOP-3330
https://issues.apache.org/jira/browse/SQOOP-3330


Repository: sqoop-trunk


Description
---

When adding --append to Sqoop directory import with 
-Dmapreduce.output.basename, all files will be ignored, which end up nothing 
being imported. See below DEBUG output:

{code}
sqoop import -Dmapreduce.output.basename="eric-test" --connect 
jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test 
--target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose 
--append

18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-0 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-1 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-2 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 
14935e396acc4ea7b9a6236c66064c9b_test
{code}

This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"

https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46


Diffs (updated)
-

  src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
  src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 


Diff: https://reviews.apache.org/r/67424/diff/4/

Changes: https://reviews.apache.org/r/67424/diff/3-4/


Testing
---

Added test case + manual test in VM


Thanks,

Eric Lin

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-07-12 Thread Eric Lin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542527#comment-16542527
 ] 

Eric Lin commented on SQOOP-3042:
-

Thanks [~vasas] for you help to review and resolve this JIRA, much appreciated!!

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3341) Oracle Data Connector Doesn't Support Case Sensitive Table Names

2018-07-12 Thread Shawn Weeks (JIRA) 
Shawn Weeks created SQOOP-3341:
--

 Summary: Oracle Data Connector Doesn't Support Case Sensitive 
Table Names
 Key: SQOOP-3341
 URL: https://issues.apache.org/jira/browse/SQOOP-3341
 Project: Sqoop
  Issue Type: Bug
  Components: connectors/oracle
Affects Versions: 1.4.6
Reporter: Shawn Weeks


Currently the Oracle Data Connector uppercases table names before querying the 
data dictionary for columns causing it to fail when you try to import a 
lowercase table name. This is probably caused by OraOopUtilities upper casing 
the table names under decodeOracleTableName. Ideally we would just accept the 
case inputed or try to detect the double quotes around the table name. From the 
debug logs you end up with something like this. The message implies some sort 
of double quoting will fix the issue but I haven't discovered any combination 
that does.

 
{code:java}
The Oracle table context has been derived from:
oracleConnectionUserName = someuser
tableStr = STAGE.blah_201612160955
as:
owner : STAGE
table : BLAH_201612160955

Data Connector for Oracle and Hadoop will not process this Sqoop connection, as 
the Oracle user STAGE does not own a table named BLAH_201612160955.
Please prefix the table name with the owner.
Note: You may need to double-quote the owner and/or table name.
E.g. sqoop ... --username someuser --table STAGE.BLAH_201612160955{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename

2018-07-12 Thread Szabolcs Vasas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/#review206012
---


Fix it, then Ship it!




Hi Eric,

Thank you for your patch, I have found a small thing only, please fix it and 
then we can go ahead and commit it.

Szabolcs


src/test/org/apache/sqoop/TestAppendUtils.java
Lines 331 (patched)


Nit: can you please use generic List here, otherwise the compiler gives a 
warning:
List args = new ArrayList<>();


- Szabolcs Vasas


On July 12, 2018, 8:11 a.m., Eric Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67424/
> ---
> 
> (Updated July 12, 2018, 8:11 a.m.)
> 
> 
> Review request for Sqoop and Boglarka Egyed.
> 
> 
> Bugs: SQOOP-3330
> https://issues.apache.org/jira/browse/SQOOP-3330
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> When adding --append to Sqoop directory import with 
> -Dmapreduce.output.basename, all files will be ignored, which end up nothing 
> being imported. See below DEBUG output:
> 
> {code}
> sqoop import -Dmapreduce.output.basename="eric-test" --connect 
> jdbc:mysql://mysql-host.com/test --username root --password 'root' --table 
> test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' 
> --verbose --append
> 
> 18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-0 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-1 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-2 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 
> 14935e396acc4ea7b9a6236c66064c9b_test
> {code}
> 
> This is due to AppendUtils only recognizes file name starts with 
> "part.*-([0-9]"
> 
> https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
>   src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 
> 
> 
> Diff: https://reviews.apache.org/r/67424/diff/3/
> 
> 
> Testing
> ---
> 
> Added test case + manual test in VM
> 
> 
> Thanks,
> 
> Eric Lin
> 
>

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-07-12 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541770#comment-16541770
 ] 

ASF subversion and git services commented on SQOOP-3042:


Commit 0cfbf56713f7574568ea3754f6854e82f5540954 in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=0cfbf56 ]

SQOOP-3042: Sqoop does not clear compile directory under 
/tmp/sqoop-/compile automatically

(Eric Lin via Szabolcs Vasas)


> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67424: SQOOP-3330 - Sqoop --append does not work with -Dmapreduce.output.basename

2018-07-12 Thread Eric Lin via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67424/
---

(Updated July 12, 2018, 8:11 a.m.)


Review request for Sqoop and Boglarka Egyed.


Changes
---

Remove "out" from .gitignore


Bugs: SQOOP-3330
https://issues.apache.org/jira/browse/SQOOP-3330


Repository: sqoop-trunk


Description
---

When adding --append to Sqoop directory import with 
-Dmapreduce.output.basename, all files will be ignored, which end up nothing 
being imported. See below DEBUG output:

{code}
sqoop import -Dmapreduce.output.basename="eric-test" --connect 
jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test 
--target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose 
--append

18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-0 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-1 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-2 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 
14935e396acc4ea7b9a6236c66064c9b_test
{code}

This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"

https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46


Diffs (updated)
-

  src/java/org/apache/sqoop/util/AppendUtils.java fa852804 
  src/test/org/apache/sqoop/TestAppendUtils.java f14fc6ad 


Diff: https://reviews.apache.org/r/67424/diff/3/

Changes: https://reviews.apache.org/r/67424/diff/2-3/


Testing
---

Added test case + manual test in VM


Thanks,

Eric Lin

[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-07-12 Thread Eric Lin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541296#comment-16541296
 ] 

Eric Lin commented on SQOOP-3042:
-

uploading SQOOP-3042.9.patch, which includes test cases and updated docs.

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

2018-07-12 Thread Eric Lin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Lin updated SQOOP-3042:

Attachment: SQOOP-3042.9.patch

> Sqoop does not clear compile directory under /tmp/sqoop-/compile 
> automatically
> 
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
>  Labels: patch
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch, 
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch, 
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left 
> behind on disk, so anyone can check those JAVA files to see the schema of 
> those tables that Sqoop has been interacting with. By default, the directory 
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see 
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
>   hashDir = new File(baseDir, RandomHash.generateMD5String());
>   while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
>   }
>   if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
>   }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)