Re: Review Request 61615: Add Sqoop1 (import + --incremental + --check-column) support for functions/expressions
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61615/
---
(Updated July 5, 2018, 8:56 a.m.)
Review request for Sqoop and Boglarka Egyed.
Changes
---
uploaded latest patch based on review and merge with latest trunk
Bugs: SQOOP-3186
https://issues.apache.org/jira/browse/SQOOP-3186
Repository: sqoop-trunk
Description
---
Add Sqoop1 (import + --incremental + --check-column) support for
functions/expressions, for example:
Example
sqoop import \
--connect $MYCONN --username $MYUSER --password $MYPSWD \
--table T1 --target-dir /path/directory --merge-key C1 \
--incremental lastmodified --last-value '2017-01-01 00:00:00.0' \
--check-column nvl(C4,to_date('2017-01-01 00:00:00')
Diffs (updated)
-
src/java/org/apache/sqoop/SqoopOptions.java 3a19aeac
src/java/org/apache/sqoop/tool/BaseSqoopTool.java 8d318327
src/java/org/apache/sqoop/tool/ImportTool.java 25c3f703
src/test/org/apache/sqoop/mapreduce/TestImportJob.java 941acb6d
Diff: https://reviews.apache.org/r/61615/diff/2/
Changes: https://reviews.apache.org/r/61615/diff/1-2/
Testing
---
Manual testing, still working on adding test cases.
This is first iteration, so code still need to be refined. I will update to
latest trunk once suggestions are made.
Thanks,
Eric Lin
[jira] [Updated] (SQOOP-3186) Add Sqoop1 (import + --incremental + --check-column) support for functions/expressions
[
https://issues.apache.org/jira/browse/SQOOP-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Lin updated SQOOP-3186:
Attachment: SQOOP-3186.2.patch
> Add Sqoop1 (import + --incremental + --check-column) support for
> functions/expressions
> --
>
> Key: SQOOP-3186
> URL: https://issues.apache.org/jira/browse/SQOOP-3186
> Project: Sqoop
> Issue Type: Improvement
>Reporter: Markus Kemper
>Assignee: Eric Lin
>Priority: Major
> Attachments: SQOOP-3186.2.patch, SQOOP-3186.patch
>
>
> Add Sqoop1 (import + --incremental + --check-column) support for
> functions/expressions, for example:
> *Example*
> {noformat}
> sqoop import \
> --connect $MYCONN --username $MYUSER --password $MYPSWD \
> --table T1 --target-dir /path/directory --merge-key C1 \
> --incremental lastmodified --last-value '2017-01-01 00:00:00.0' \
> --check-column nvl(C4,to_date('2017-01-01 00:00:00')
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
[
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533387#comment-16533387
]
Eric Lin commented on SQOOP-3042:
-
Added patch based on latest trunk, waiting for review.
> Sqoop does not clear compile directory under /tmp/sqoop-/compile
> automatically
>
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
> Labels: patch
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch,
> SQOOP-3042.4.patch, SQOOP-3042.5.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left
> behind on disk, so anyone can check those JAVA files to see the schema of
> those tables that Sqoop has been interacting with. By default, the directory
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> }
> if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
> }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 54528: SQOOP-3042 - Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54528/
---
(Updated July 5, 2018, 8:03 a.m.)
Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
Changes
---
Updated to the latest trunk. Unable to add test case due to deleteOnExit won't
happen until test finishes.
Bugs: SQOOP-3042
https://issues.apache.org/jira/browse/SQOOP-3042
Repository: sqoop-trunk
Description
---
After running sqoop, all the temp files generated by ClassWriter are left
behind on disk, so anyone can check those JAVA files to see the schema of those
tables that Sqoop has been interacting with. By default, the directory is under
/tmp/sqoop-/compile.
In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
that we did add "deleteOnExit" on the temp dir:
for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
hashDir = new File(baseDir, RandomHash.generateMD5String());
while (hashDir.exists()) {
hashDir = new File(baseDir, RandomHash.generateMD5String());
}
if (hashDir.mkdirs()) {
// We created the directory. Use it.
// If this directory is not actually filled with files, delete it
// when the JVM quits.
hashDir.deleteOnExit();
break;
}
}
However, I believe it failed to delete due to directory is not empty.
Diffs (updated)
-
.gitignore 68cbe287
src/java/org/apache/sqoop/SqoopOptions.java 3a19aeac
src/java/org/apache/sqoop/orm/ClassWriter.java a4a768af
src/java/org/apache/sqoop/orm/CompilationManager.java 6590cacc
src/java/org/apache/sqoop/tool/BaseSqoopTool.java 8d318327
Diff: https://reviews.apache.org/r/54528/diff/3/
Changes: https://reviews.apache.org/r/54528/diff/2-3/
Testing
---
I have tested manually. I have checked with a couple of other Java developers
and it turned out that it is not easy to add test for deleteOnExit, so I did
not add any test cases. The code path I changed does not seem to have test
coverage either. Let me know if I am wrong.
Thanks
Thanks,
Eric Lin
