[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
[
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267377#comment-16267377
]
Jean-Marc Spaggiari commented on SQOOP-3042:
Any chance to get that rebased and commited?
> Sqoop does not clear compile directory under /tmp/sqoop-/compile
> automatically
>
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
> Labels: patch
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch,
> SQOOP-3042.4.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left
> behind on disk, so anyone can check those JAVA files to see the schema of
> those tables that Sqoop has been interacting with. By default, the directory
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> }
> if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
> }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: Review Request 61882: SQOOP-3215 : sqoop import to hive table as formats(avro, parquet)
> On Nov. 16, 2017, 1:43 p.m., Szabolcs Vasas wrote: > > Hi Sandish, > > > > I have took another look at your patch and I think there is some confusion > > here. > > > > * The JIRA description says that create-hive-table tool needs to be > > improved to be able to create Parquet and Avro tables too (schema only). > > The problem is that there is a --create-hive-table option as well which is > > a totally different thing. The --create-hive-table option gets mapped to > > org.apache.sqoop.SqoopOptions#failIfHiveTableExists boolean value (which is > > a much better description) and it means that Sqoop job gets aborted if the > > Hive table already exist. I think your patch tries to address the > > --create-hive-table option so it should be done under a different JIRA. > > * Based on the code you try to include 2 changes in this patch and I think > > they should be separated. > > * The first thing was the Parquet Hive table creation changes in > > TableDefWriter. The problem is that I think this code never gets executed > > since org.apache.sqoop.hive.HiveImport#importTable never gets invoked if we > > import into Parquet files. See: org/apache/sqoop/tool/ImportTool.java:539 > > * The other thing is the Avro support for Hive tables. I think this part > > still needs more testing with possibly all the supported data types and > > codecs. > > > > Regards, > > Szabolcs Hi @vasas, Thanks for review, I will create two seperate JIRA issues for this patch. One for "Sqoop import parquet support" and "Sqoop import Avro support" - Sandish Kumar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61882/#review191177 --- On Oct. 23, 2017, 12:30 p.m., Sandish Kumar HN wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61882/ > --- > > (Updated Oct. 23, 2017, 12:30 p.m.) > > > Review request for Sqoop and Anna Szonyi. > > > Bugs: SQOOP-3215 > https://issues.apache.org/jira/browse/SQOOP-3215 > > > Repository: sqoop-trunk > > > Description > --- > > sqoop import and create hive table as formats(avro,parquet) > > > Diffs > - > > src/java/org/apache/sqoop/hive/TableDefWriter.java deec32d6 > src/java/org/apache/sqoop/tool/BaseSqoopTool.java 6a4dcb09 > src/test/com/cloudera/sqoop/hive/TestHiveImport.java a624f52b > testdata/hive/scripts/normalHiveImportAvro.q PRE-CREATION > > > Diff: https://reviews.apache.org/r/61882/diff/2/ > > > Testing > --- > > testNormalHiveImportAsAvro, testNormalHiveImportAsParquet with table creation > checks. > > > Thanks, > > Sandish Kumar HN > >
Re: Review Request 64034: Sqoop must not log database passwords
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64034/ --- (Updated Nov. 27, 2017, 9:55 a.m.) Review request for Sqoop. Bugs: SQOOP-3257 https://issues.apache.org/jira/browse/SQOOP-3257 Repository: sqoop-trunk Description --- If sqoop.metastore.client.record.password property is set to true then Sqoop stores the database password in the metastore. The problem is that in this case the password can be logged by a couple of classes which needs to be avoided. Diffs (updated) - src/docs/user/saved-jobs.txt d21df02 src/java/org/apache/sqoop/SqoopJobDataPublisher.java 93be02f src/java/org/apache/sqoop/SqoopOptions.java 587d4e1 src/java/org/apache/sqoop/metastore/GenericJobStorage.java 9e1b18b src/java/org/apache/sqoop/metastore/PasswordRedactor.java PRE-CREATION src/java/org/apache/sqoop/tool/JobTool.java dbe8934 src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java bd911f2 src/test/org/apache/sqoop/metastore/PasswordRedactorTest.java PRE-CREATION Diff: https://reviews.apache.org/r/64034/diff/2/ Changes: https://reviews.apache.org/r/64034/diff/1-2/ Testing --- Unit tests and third party tests are executed. Thanks, Szabolcs Vasas
