Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On April 23, 2018, 10:33 a.m., Fero Szabo wrote:
> > Hi Anna,
> > 
> > I couldn't apply the patch cleanly, I ran into the following error message:
> > 
> > fero-MBP:sqoop ferencszabo$ git apply ~/Downloads/SQOOP-3052-5.patch 
> > error: missing binary patch data for 'gradle/wrapper/gradle-wrapper.jar'
> > error: binary patch does not apply to 'gradle/wrapper/gradle-wrapper.jar'
> > error: gradle/wrapper/gradle-wrapper.jar: patch does not apply
> > 
> > Managed to apply it with the following command:
> > patch -p1 -i  ~/Downloads/SQOOP-3052-5.patch
> > 
> > Also, gradle was missing from my system, so I had to install it and then 
> > gradle/wrapper/gradle-wrapper.jar was probably downloaded after I executed 
> > 'gradle wrapper' or './gradlew wrapper'
> > 
> > I'm not sure how to fix these though. Did I go through the "correct" way to 
> > install gradle? Maybe I should have invoked './gradlew wrapper' right after 
> > applying the patch. Would that work without gradle installed on the system?
> > 
> > Anyway thanks for the patch, it would be really cool to use gradle in Sqoop!
> 
> Anna Szonyi wrote:
> Hi Fero,
> 
> Thanks for testing, I haven't rebased to the latest version of trunk, 
> which might be the reason it doesn't apply cleanly? I tested it when I 
> created the patch on the current (at that time) trunk, but I will definitely 
> double-check this.
> 
> The reason we are adding the gradle wrapper jar and the gradlew and 
> gradlew.bat files is that you shouldn't have been required to separately 
> install gradle - just execute ./gradlew wrapper.
> 
> Thanks,
> Anna

I've regenerated the diff with the --binary flag as well, which seems to be the 
solution for your apply issues, the rest of the issues stem from this.


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review201727
---


On April 24, 2018, 2:23 p.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated April 24, 2018, 2:23 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
>   testdata/hcatalog/conf/hive-site.xml 
> edac7aa9087a84b7a0c660907794adae684ae313 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/10/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On April 23, 2018, 3:06 p.m., Szabolcs Vasas wrote:
> > Hi Anna,
> > 
> > Thank you for fixing the HiveImportTest problem, it works fine now.
> > I have added a few more comments, there are a few test cases failing now 
> > because of SQOOP-3309 but it is easy to fix it.
> > 
> > However relnotes and release tasks fail on my side are they supposed to 
> > work with some extra parameters?

Would you mind including the error message? It is supposed to skip this step in 
case the version is a SNAPSHOT version. If you specify -Pversion it should run 
(it should suceed in both cases, but skip execution in the snapshot case).


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review201738
-------


On April 24, 2018, 2:23 p.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated April 24, 2018, 2:23 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
>   testdata/hcatalog/conf/hive-site.xml 
> edac7aa9087a84b7a0c660907794adae684ae313 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/10/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated April 24, 2018, 2:23 p.m.)


Review request for Sqoop.


Changes
---

changed per Szabi's comments and regenerated with --binary per Fero's issue.


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
  COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
  build.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/customUnixStartScript.txt PRE-CREATION 
  gradle/customWindowsStartScript.txt PRE-CREATION 
  gradle/sqoop-package.gradle PRE-CREATION 
  gradle/sqoop-version-gen.gradle PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 
  src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
  testdata/hcatalog/conf/hive-site.xml edac7aa9087a84b7a0c660907794adae684ae313 


Diff: https://reviews.apache.org/r/66067/diff/10/

Changes: https://reviews.apache.org/r/66067/diff/9-10/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated April 23, 2018, 10:40 a.m.)


Review request for Sqoop.


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
  COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
  build.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/customUnixStartScript.txt PRE-CREATION 
  gradle/customWindowsStartScript.txt PRE-CREATION 
  gradle/sqoop-package.gradle PRE-CREATION 
  gradle/sqoop-version-gen.gradle PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 
  src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
  testdata/hcatalog/conf/hive-site.xml edac7aa9087a84b7a0c660907794adae684ae313 


Diff: https://reviews.apache.org/r/66067/diff/9/

Changes: https://reviews.apache.org/r/66067/diff/8-9/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On April 23, 2018, 10:33 a.m., Fero Szabo wrote:
> > Hi Anna,
> > 
> > I couldn't apply the patch cleanly, I ran into the following error message:
> > 
> > fero-MBP:sqoop ferencszabo$ git apply ~/Downloads/SQOOP-3052-5.patch 
> > error: missing binary patch data for 'gradle/wrapper/gradle-wrapper.jar'
> > error: binary patch does not apply to 'gradle/wrapper/gradle-wrapper.jar'
> > error: gradle/wrapper/gradle-wrapper.jar: patch does not apply
> > 
> > Managed to apply it with the following command:
> > patch -p1 -i  ~/Downloads/SQOOP-3052-5.patch
> > 
> > Also, gradle was missing from my system, so I had to install it and then 
> > gradle/wrapper/gradle-wrapper.jar was probably downloaded after I executed 
> > 'gradle wrapper' or './gradlew wrapper'
> > 
> > I'm not sure how to fix these though. Did I go through the "correct" way to 
> > install gradle? Maybe I should have invoked './gradlew wrapper' right after 
> > applying the patch. Would that work without gradle installed on the system?
> > 
> > Anyway thanks for the patch, it would be really cool to use gradle in Sqoop!

Hi Fero,

Thanks for testing, I haven't rebased to the latest version of trunk, which 
might be the reason it doesn't apply cleanly? I tested it when I created the 
patch on the current (at that time) trunk, but I will definitely double-check 
this.

The reason we are adding the gradle wrapper jar and the gradlew and gradlew.bat 
files is that you shouldn't have been required to separately install gradle - 
just execute ./gradlew wrapper.

Thanks,
Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review201727
---


On April 18, 2018, 3:10 p.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated April 18, 2018, 3:10 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
>   testdata/hcatalog/conf/hive-site.xml 
> edac7aa9087a84b7a0c660907794adae684ae313 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/9/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

t; > org.apache.sqoop.hive.TestHiveImport > testAppendHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImport PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNormalHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportWithBadPartitionKey PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > 
> > testCreateOverwriteHiveImportAsParquet STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > 
> > testCreateOverwriteHiveImportAsParquet PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testImportHiveWithPartitions PASSED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNumeric STARTED
> > 
> > org.apache.sqoop.hive.TestHiveImport > testNumeric PASSED
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; 
> > support was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option 
> > MaxPermSize=512m; support was removed in 8.0
> > objc[8]: Class JavaLaunchHelper is implemented in both 
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java 
> > and 
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
> >  One of the two will be used. Which one is undefined.
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; 
> > support was removed in 8.0
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option 
> > MaxPermSize=512m; support was removed in 8.0
> > objc[9]: Class JavaLaunchHelper is implemented in both 
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java 
> > and 
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib.
> >  One of the two will be used. Which one is undefined.
> > ...
> > ...
> > 
> > BUILD SUCCESSFUL
> > 
> > Total time: 2 mins 54.654 secs
> > 
> > 
> > I am not sure what happens here, my guess is that a new JVM is started for 
> > every test case even if they are not executed. Have you experienced this 
> > problem too?
> 
> Anna Szonyi wrote:
> I've had similar issues running with the --tests option, I usually use 
> -Dtest.single, unfortunately if you have a buildSrc folder in your project 
> that seems to mess everything up for test.single, so I can rename that 
> directory to solve this for now. Oddly enough it doesn't affect 
> thirdParty.single.
> Maybe in the long run we can also try to make test runs independent, so 
> we don't have to use forkevery 1...

Hey Szabi,

Please try with test.single with the new patch - you shouldn't face the same 
issues!

Thanks,
Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199896
---


On April 13, 2018, 9:06 a.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated April 13, 2018, 9:06 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/7/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On March 28, 2018, 11:02 a.m., daniel voros wrote:
> > Hey Anna,
> > 
> > I've experimented with running the gradle build in a clean dockerized 
> > environment and found some minor issues:
> >  1) All dependencies are downloaded from jcenter, despite having central 
> > repository defined in build.gradle. This might be a result of my current 
> > setup, could you please confirm?
> >  2) Deprecation warnings for '<<' task doLast syntax ("Deprecation warning: 
> > The Task.leftShift(Closure) method has been deprecated...")
> >  3) SqoopVersion.java generation happens during task definition and not in 
> > action (missing doLast?).
> >  4) `relnotes` task fails if version is SNAPSHOT with: "A problem occurred 
> > starting process 'command 'cd''".
> >  5) The `release` task prints the path of tar and rat report but they're 
> > incorrect. (I've specified version on the command line with 
> > "-Pversion=1.5.0")
> >  6) `ant releaseaudit` now lists gradle files as errors
> > 
> > I've corrected 2,3,4,5,6 in this commit: 
> > https://github.com/dvoros/sqoop/commit/47e361829b1004bdedd6f5c223332e3fb8b85696
> > 
> > What's the reasoning behind using Gradle 3.5.1? Shouldn't we use 4.x? (I've 
> > successfully executed a simple build with 4.6)
> > 
> > Regards,
> > Daniel

Hi Daniel,

Thanks for testing and the suggestions, I've incorporated your feedback! 

Wrt gradle 3.5.1: for myself and some of the people testing this on their local 
machine tests weren't running with gradle 4.x+ and the fix wasn't trivial, so 
we decided to nail it down to 3.5 and increase it later, as it will probably 
result in code change as well.

#1 it uses jcenter for me as well by default, it might be because jcenter 
became the new default for gradle, as far as I can tell for the following 
reasons: 
https://blog.bintray.com/2015/02/09/android-studio-migration-from-maven-central-to-jcenter/,
 though tbh this isn't the expected behaviour - does everyone else experience 
this also?
#2 Oddly enough it behaves slightly differently for me, but not as 
significantly as --tests and test.single so I changed them all 
#3 I've added doLast, but it doesn't change the order, not sure if it's worth 
further investigation.
#5 I missed this, I will correct it, thanks!

Thanks,
Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review200102
---


On April 13, 2018, 9:06 a.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated April 13, 2018, 9:06 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
>   COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
>   build.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/customUnixStartScript.txt PRE-CREATION 
>   gradle/customWindowsStartScript.txt PRE-CREATION 
>   gradle/sqoop-package.gradle PRE-CREATION 
>   gradle/sqoop-version-gen.gradle PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
>   src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/7/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated April 13, 2018, 9:06 a.m.)


Review request for Sqoop.


Changes
---

Changes based on review feedback


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28731e613607c208824443d1edf256d9c8a 
  COMPILING.txt 3b82250488256871352056e9061ad08fabbd7fc5 
  build.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/customUnixStartScript.txt PRE-CREATION 
  gradle/customWindowsStartScript.txt PRE-CREATION 
  gradle/sqoop-package.gradle PRE-CREATION 
  gradle/sqoop-version-gen.gradle PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 
  src/scripts/rat-violations.sh 1cfbc1502b24dd1b8b7e7ce21f0b5d1880c06556 


Diff: https://reviews.apache.org/r/66067/diff/7/

Changes: https://reviews.apache.org/r/66067/diff/6-7/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

24)
> at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at 
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:147)
> at 
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:129)
> at 
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:46)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ```
> 
> I think they were relying on some system properties that we had defined 
> in build.xml but not in build.gradle (e.g. ms.datatype.test.data.file.export).
> 
> Other than that, results are the same, both for regular and third-party 
> tests.
> 
> Regards,
> Daniel

Hey guys,

Thanks for checking this I missed it. There are some missing sql server 
properties. I have added them and verified that they are now running. 

Thanks,
Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199930
---


On March 23, 2018, 10:28 a.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated March 23, 2018, 10:28 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28 
>   COMPILING.txt 3b82250 
>   build.gradle PRE-CREATION 
>   buildSrc/customUnixStartScript.txt PRE-CREATION 
>   buildSrc/customWindowsStartScript.txt PRE-CREATION 
>   buildSrc/sqoop-package.gradle PRE-CREATION 
>   buildSrc/sqoop-version-gen.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/6/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated March 23, 2018, 10:28 a.m.)


Review request for Sqoop.


Changes
---

Apologies for the barrage of changes - fixed messed up diff


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28 
  COMPILING.txt 3b82250 
  build.gradle PRE-CREATION 
  buildSrc/customUnixStartScript.txt PRE-CREATION 
  buildSrc/customWindowsStartScript.txt PRE-CREATION 
  buildSrc/sqoop-package.gradle PRE-CREATION 
  buildSrc/sqoop-version-gen.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 


Diff: https://reviews.apache.org/r/66067/diff/6/

Changes: https://reviews.apache.org/r/66067/diff/5-6/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated March 23, 2018, 10:19 a.m.)


Review request for Sqoop.


Changes
---

Fixed diff vs. trunk


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28 
  COMPILING.txt 3b82250 
  src/docs/user/import.txt e91a5a8 


Diff: https://reviews.apache.org/r/66067/diff/5/

Changes: https://reviews.apache.org/r/66067/diff/4-5/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On March 23, 2018, 9:39 a.m., Szabolcs Vasas wrote:
> > Hi Anna,
> > 
> > Thank you for replying so quickly!
> > The changes look good in RB but I could no apply it in my repo, I get the 
> > following error:
> > 
> > error: patch failed: COMPILING.txt:403
> > error: COMPILING.txt: patch does not apply
> > error: cannot apply binary patch to 'gradle/wrapper/gradle-wrapper.jar' 
> > without full index line
> > error: gradle/wrapper/gradle-wrapper.jar: patch does not apply

Apologies, please try again. I ran the diff with diff-index --binary and it 
applies for me.


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199871
---


On March 23, 2018, 10:05 a.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated March 23, 2018, 10:05 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28 
>   COMPILING.txt 86be509 
>   build.gradle PRE-CREATION 
>   buildSrc/customUnixStartScript.txt PRE-CREATION 
>   buildSrc/customWindowsStartScript.txt PRE-CREATION 
>   buildSrc/sqoop-package.gradle PRE-CREATION 
>   buildSrc/sqoop-version-gen.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/4/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated March 23, 2018, 10:05 a.m.)


Review request for Sqoop.


Changes
---

ran the diff with diff-index --binary


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28 
  COMPILING.txt 86be509 
  build.gradle PRE-CREATION 
  buildSrc/customUnixStartScript.txt PRE-CREATION 
  buildSrc/customWindowsStartScript.txt PRE-CREATION 
  buildSrc/sqoop-package.gradle PRE-CREATION 
  buildSrc/sqoop-version-gen.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 


Diff: https://reviews.apache.org/r/66067/diff/4/

Changes: https://reviews.apache.org/r/66067/diff/3-4/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated March 22, 2018, 8:53 p.m.)


Review request for Sqoop.


Changes
---

added sourcecompatibilityversion, do not copy .gradle to packaging, gitignore 
.gradle


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28 
  COMPILING.txt 86be509 
  build.gradle PRE-CREATION 
  buildSrc/customUnixStartScript.txt PRE-CREATION 
  buildSrc/customWindowsStartScript.txt PRE-CREATION 
  buildSrc/sqoop-package.gradle PRE-CREATION 
  buildSrc/sqoop-version-gen.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 


Diff: https://reviews.apache.org/r/66067/diff/3/

Changes: https://reviews.apache.org/r/66067/diff/2-3/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

> On March 22, 2018, 4:19 p.m., Boglarka Egyed wrote:
> > gradlew
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/66067/diff/1/?file=1982323#file1982323line1>
> >
> > License information header is missing from this file.

I don't think we can add the license header to this file, as it's the auto 
generated gradle "runner" file.


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/#review199766
-------


On March 22, 2018, 6:32 p.m., Anna Szonyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66067/
> ---
> 
> (Updated March 22, 2018, 6:32 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: Sqoop-3052
> https://issues.apache.org/jira/browse/Sqoop-3052
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
> friendly / open
> 
> 
> Diffs
> -
> 
>   .gitignore 68cbe28 
>   COMPILING.txt 86be509 
>   build.gradle PRE-CREATION 
>   buildSrc/customUnixStartScript.txt PRE-CREATION 
>   buildSrc/customWindowsStartScript.txt PRE-CREATION 
>   buildSrc/sqoop-package.gradle PRE-CREATION 
>   buildSrc/sqoop-version-gen.gradle PRE-CREATION 
>   config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
>   config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
>   config/checkstyle/checkstyle.xml PRE-CREATION 
>   gradle.properties PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
>   gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
>   gradlew PRE-CREATION 
>   gradlew.bat PRE-CREATION 
>   settings.gradle PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66067/diff/2/
> 
> 
> Testing
> ---
> 
> ran all new tasks, except for internal maven publishing
> 
> Notes:
> - To try it out you can call ./gradlew tasks --all to see all the tasks and 
> compare them to current tasks/artifacts.
> - Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
> to combine all test results into a single report.
> - Generated pom.xml now has correct dependencies/versions
> - Script generation is currently hardcoded and not based on sqoop help, as 
> previously - though added the possiblity of hooking it in later
> 
> 
> Thanks,
> 
> Anna Szonyi
> 
>

Re: Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

(Updated March 22, 2018, 6:32 p.m.)


Review request for Sqoop.


Changes
---

changed per Bogi's comment + added the missing gradle/gradle-wrapper.jar


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs (updated)
-

  .gitignore 68cbe28 
  COMPILING.txt 86be509 
  build.gradle PRE-CREATION 
  buildSrc/customUnixStartScript.txt PRE-CREATION 
  buildSrc/customWindowsStartScript.txt PRE-CREATION 
  buildSrc/sqoop-package.gradle PRE-CREATION 
  buildSrc/sqoop-version-gen.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradle/wrapper/gradle-wrapper.jar PRE-CREATION 
  gradle/wrapper/gradle-wrapper.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 


Diff: https://reviews.apache.org/r/66067/diff/2/

Changes: https://reviews.apache.org/r/66067/diff/1-2/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: SQOOP-3052.patch

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66067: SQOOP-3052: Introduce gradle-based build for Sqoop to make it more developer friendly / open

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66067/
---

Review request for Sqoop.


Bugs: Sqoop-3052
https://issues.apache.org/jira/browse/Sqoop-3052


Repository: sqoop-trunk


Description
---

SQOOP-3052: Introduce gradle based build for Sqoop to make it more developer 
friendly / open


Diffs
-

  .gitignore 68cbe28 
  COMPILING.txt 86be509 
  build.gradle PRE-CREATION 
  buildSrc/customUnixStartScript.txt PRE-CREATION 
  buildSrc/customWindowsStartScript.txt PRE-CREATION 
  buildSrc/sqoop-package.gradle PRE-CREATION 
  buildSrc/sqoop-version-gen.gradle PRE-CREATION 
  config/checkstyle/checkstyle-java-header.txt PRE-CREATION 
  config/checkstyle/checkstyle-noframes.xsl PRE-CREATION 
  config/checkstyle/checkstyle.xml PRE-CREATION 
  gradle.properties PRE-CREATION 
  gradlew PRE-CREATION 
  gradlew.bat PRE-CREATION 
  settings.gradle PRE-CREATION 


Diff: https://reviews.apache.org/r/66067/diff/1/


Testing
---

ran all new tasks, except for internal maven publishing

Notes:
- To try it out you can call ./gradlew tasks --all to see all the tasks and 
compare them to current tasks/artifacts.
- Replaced cobertura with jacoco, as it's easier/cleaner to configure, easier 
to combine all test results into a single report.
- Generated pom.xml now has correct dependencies/versions
- Script generation is currently hardcoded and not based on sqoop help, as 
previously - though added the possiblity of hooking it in later


Thanks,

Anna Szonyi

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: (was: SQOOP-3052.patch)

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: SQOOP-3052.patch

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: (was: SQOOP-3052.patch)

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: (was: SQOOP-3052.patch)

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404790#comment-16404790
 ] 

Anna Szonyi commented on SQOOP-3052:


 Hi [~maugli],

It seems I've missed your comment - apologies! The patches I attached were in a 
mostly complete state, I attached it as I got some requests around trying it 
out, but there were some files missing (script templates, etc.) and some 
further polishing to be done.

I'll attach the finished version.

Best,

Anna

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch, SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: SQOOP-3052.patch

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch, SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi reassigned SQOOP-3052:
--

Assignee: Anna Szonyi  (was: Anna Szonyi)

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3052) Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer friendly / open

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3052:
---
Attachment: SQOOP-3052.patch

> Introduce Maven/Gradle/etc. based build for Sqoop to make it more developer 
> friendly / open
> ---
>
> Key: SQOOP-3052
> URL: https://issues.apache.org/jira/browse/SQOOP-3052
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Attila Szabo
>    Assignee: Anna Szonyi
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: SQOOP-3052.patch
>
>
> The current trunk version can only be build with Ant/Ivy combination, which 
> has some painful limitations (resolve is slow / needs to be tweaked to use 
> only caches, the current profile / variable based settings are not working in 
> IDEs out of the box, the current solution does not download the related 
> sources, etc.)
> It would be nice to provide a solution, which would give the possibility for 
> the developers to choose between the nowadays well used build infrsturctures 
> (e.g. Maven, Gradle, etc.). For this solution it would be also essential to 
> keep the different build files (if there is more then one) synchronized 
> easily, and the configuration wouldn't diverege by time. Test execution has 
> to be solved also, and should cover all the available test cases.
> In this scenario:
> If we can provide one good working solution is much better, then provide 
> three different ones which become out of sync easily. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Sqoop version 1.4.7

[Anna Szonyi]

+1 to continuing the release, also a belated +1 (non-binding) for the
release candidate!

On Mon, Jan 15, 2018 at 10:49 AM, Szabolcs Vasas  wrote:

> Hi Attila,
>
> It seems RC0 has received the necessary 3 votes from PMC members, can we
> proceed with the release process?
> Please let us know if you need any help.
>
> Regards,
> Szabolcs
>
> On Thu, Jan 11, 2018 at 5:30 AM, Abraham Fine  wrote:
>
> > +1
> >
> > On Tue, Jan 9, 2018, at 17:15, Venkat Ranganathan wrote:
> > > +1
> > > Venkat
> > >
> > > On 1/9/18, 10:25 AM, "Kathleen Ting"  wrote:
> > >
> > > Thanks Attila for being the release manager.
> > >
> > > +1 on Sqoop 1.4.7 RC0.
> > >
> > > Thanks,
> > > Kate
> > >
> > > On Fri, Jan 5, 2018 at 3:17 AM, Boglarka Egyed 
> > wrote:
> > >
> > > > Hi Attila,
> > > >
> > > > Thank you for preparing this release!
> > > >
> > > > I have checked the followings on RC0:
> > > > - checksums OK
> > > > - signature OK
> > > > - LICENSE and NOTICE files OK
> > > > - compile OK
> > > > - unit tests OK
> > > > - 3rd party tests OK
> > > >
> > > > Based on this +1 from my side too.
> > > >
> > > > Many thanks,
> > > > Bogi
> > > >
> > > > On Fri, Jan 5, 2018 at 12:04 PM, Attila Szabó  >
> > wrote:
> > > >
> > > > > Dear Sqoop community,
> > > > >
> > > > > I'd like to kindly remind and ask you to vote on Sqoop 1.4.7
> > RC0, if you
> > > > > would have some spare time.
> > > > >
> > > > > Thanks in advance,
> > > > > Attila
> > > > >
> > > > >
> > > > > On Dec 21, 2017 6:44 PM, "Attila Szabó" 
> > wrote:
> > > > >
> > > > > Hey Szabi,
> > > > >
> > > > > AWESOME!
> > > > >
> > > > > Thanks for validating my efforts!!
> > > > >
> > > > > Cheers,
> > > > > Attila
> > > > >
> > > > > On Dec 21, 2017 6:36 PM, "Szabolcs Vasas" 
> > wrote:
> > > > >
> > > > > > Hi Attila,
> > > > > >
> > > > > > Thank you for updating the artifacts!
> > > > > >
> > > > > > I have executed a few test commands, they worked fine.
> > > > > > The unit tests and the third party tests are passing.
> > > > > > The signature and hash files are correct.
> > > > > >
> > > > > > From my side it looks good, +1
> > > > > >
> > > > > > Thank you for preparing this release!
> > > > > >
> > > > > > Szabolcs
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 21, 2017 at 4:29 PM, Attila Szabó <
> > mau...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > I've rebuilt the RC0 with the proper line endings (not crlf
> > git
> > > > > options).
> > > > > > > Updated the tar.gz .asc and .md5 files under the specified
> > location.
> > > > > > >
> > > > > > > If you need any further changes please notify me.
> > > > > > >
> > > > > > > On Thu, Dec 21, 2017 at 4:08 PM, Attila Szabó <
> > mau...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Szabi,
> > > > > > > >
> > > > > > > > Many thanks! It looks like the core.autocrlf global git
> > config
> > > > > tricked
> > > > > > > > me... Nice catch!
> > > > > > > > What would you prefer:
> > > > > > > > Just rebuild the RC0, or retag the same state as RC1 and
> > do a
> > > > totally
> > > > > > new
> > > > > > > > RC1 build and vote? (I did not see rules for this)
> > > > > > > >
> > > > > > > > The proper link for the keys file:
> > > > > > > > http://www.apache.org/dist/sqoop/KEYS
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Attila
> > > > > > > >
> > > > > > > > On Thu, Dec 21, 2017 at 3:21 PM, Szabolcs Vasas <
> > va...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Attila,
> > > > > > > >>
> > > > > > > >> I did not have time to run the full testing yet but it
> > seems that
> > > > > the
> > > > > > > >> scripts in the bin directory have Microsoft-style line
> > endings so
> > > > I
> > > > > am
> > > > > > > not
> > > > > > > >> able to start them.
> > > > > > > >> Also http://www.apache.org/dist/incubator/sqoop/KEYS
> > seems to be
> > > > > > > broken,
> > > > > > > >> can you please update the link?
> > > > > > > >>
> > > > > > > >> Thanks and regards,
> > > > > > > >> Szabolcs
> > > > > > > >>
> > > > > > > >> On Tue, Dec 19, 2017 at 11:03 PM, Attila Szabó <
> > mau...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > This is the first release for Apache Sqoop, version
> > 1.4.7
> > > > > > > >> >
> > > > > > > >> > *** Please cast your vote by [3 working days after
> > sending] ***
> > > > > > > >> >
> > > > > > > >> > The list of 

Re: Review Request 65017: Removing com.cloudera.sqoop packages

[Anna Szonyi via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65017/#review195227
---


Ship it!




Hi Szabi,

This is great, it helps a lot with the readability!

Thanks,
Anna

- Anna Szonyi


On Jan. 9, 2018, 2:25 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65017/
> ---
> 
> (Updated Jan. 9, 2018, 2:25 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3273
> https://issues.apache.org/jira/browse/SQOOP-3273
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This patch contains all the changes related to this JIRA.
> I have also created a pull request (https://github.com/apache/sqoop/pull/42) 
> because I think it is much easier to review it there since almost all of the 
> class removals were committed separately in my own Sqoop fork.
> 
> 
> Diffs
> -
> 
>   build.xml 2094bae8dde5fdaa40a7a1f86858cc79be198363 
>   conf/sqoop-site-template.xml 2182da3c88d3c6ff7dddea9f30f363435619285c 
>   ivy.xml d1530384efdecb45bd5b8ef802b60325da1dd400 
>   src/docs/dev/api-reference.txt 80453eec537a26f65e3e24b3d4da8741aceecb5b 
>   src/docs/sip/sip-4.txt 966412577126878065308324330342c6d0666f58 
>   src/java/com/cloudera/sqoop/ConnFactory.java 
> f75cb4e97e5b2febc21e59a6a6b827af010aa003 
>   src/java/com/cloudera/sqoop/Sqoop.java 
> 8ec9f8f044d10e8e071ac8a69c5860efc891f5fb 
>   src/java/com/cloudera/sqoop/SqoopOptions.java 
> 0863ef6dd71a67a5811d73bd0ef964e65e0e7afc 
>   src/java/com/cloudera/sqoop/cli/RelatedOptions.java 
> c6453b726b55fd87e1b596c3e537390582ece0a9 
>   src/java/com/cloudera/sqoop/cli/SqoopParser.java 
> 0c6d84e04c72c832a4b7160cdfffe7ebdf0c9284 
>   src/java/com/cloudera/sqoop/cli/ToolOptions.java 
> 0b5da6c3854543ceb1f1f11448d9f9bf8cba159d 
>   src/java/com/cloudera/sqoop/config/ConfigurationConstants.java 
> 72e7df133ec8bfd7be9f9006fdf8d399885ef230 
>   src/java/com/cloudera/sqoop/config/ConfigurationHelper.java 
> 0870497754a79b54c0a9338567e9af6adf931fca 
>   src/java/com/cloudera/sqoop/hbase/HBasePutProcessor.java 
> 425b0f46be7fb396ea92837c046ed9f8fbb129b1 
>   src/java/com/cloudera/sqoop/hbase/HBaseUtil.java 
> 6f1fba7d590e02c2b5f63b6e95e1ab5ad55c2e3d 
>   src/java/com/cloudera/sqoop/hbase/PutTransformer.java 
> b8ceab7782ce7bbb5c6e27eda6619ab086a6d28c 
>   src/java/com/cloudera/sqoop/hbase/ToStringPutTransformer.java 
> 0c55b21ffbe2012705714c53bd33a8e672cc5ae0 
>   src/java/com/cloudera/sqoop/hive/HiveImport.java 
> db1c1b323b782a5a7c870e9802781a73f5617079 
>   src/java/com/cloudera/sqoop/hive/HiveTypes.java 
> e67f6b890bc25c19656c6bde99aeebc592b4d45f 
>   src/java/com/cloudera/sqoop/hive/TableDefWriter.java 
> d19e01863f54f9c013e01a1f422c6dbd711ca7e0 
>   src/java/com/cloudera/sqoop/io/CodecMap.java 
> 647cedcd2d3a8e5d4836f47f779ece8a78f0bb05 
>   src/java/com/cloudera/sqoop/io/FixedLengthInputStream.java 
> 806af22c01ed93f80ee32d4b5c1c203fe2dcdb5d 
>   src/java/com/cloudera/sqoop/io/LobFile.java 
> 905c68f64fd5b45665336cc0f27c5e9ace7eacdd 
>   src/java/com/cloudera/sqoop/io/LobReaderCache.java 
> 89d31d31aaca0f60d0042a5e3472b12dac01e0c9 
>   src/java/com/cloudera/sqoop/io/NamedFifo.java 
> e27d2c4f857bb806f2686d6d447aa0eb02fc97a3 
>   src/java/com/cloudera/sqoop/io/SplittableBufferedWriter.java 
> ef9285a62c1c35f48bdfae124a22c94a4e19b6d0 
>   src/java/com/cloudera/sqoop/io/SplittingOutputStream.java 
> ab81042b48f083dc10b5b81b5b5a53505106f26e 
>   src/java/com/cloudera/sqoop/io/UnsupportedCodecException.java 
> 4d8225a3d42477a2ac8c854d6f0bd303a99c4681 
>   src/java/com/cloudera/sqoop/lib/BigDecimalSerializer.java 
> 2ae89c24f053d6176e642f547e483c6cb4cba168 
>   src/java/com/cloudera/sqoop/lib/BlobRef.java 
> b3d5341e3e29289f985d15d259a167b7b7db7002 
>   src/java/com/cloudera/sqoop/lib/BooleanParser.java 
> ab97cf0a3156f3f8f09aa2ff7c427f8014f42e21 
>   src/java/com/cloudera/sqoop/lib/ClobRef.java 
> a328f23bfee882c850a4a2aa63e3ac364307f7a9 
>   src/java/com/cloudera/sqoop/lib/DelimiterSet.java 
> 6de90ad9419610ab9e3aabe96d1a3b13ca4d768e 
>   src/java/com/cloudera/sqoop/lib/FieldFormatter.java 
> 45fb81f40d86f8a9ff8ba8f5d5334dfbe3fed644 
>   src/java/com/cloudera/sqoop/lib/FieldMapProcessor.java 
> 3f215401f91f5934e4ac6b3830b2986032e336dd 
>   src/java/com/cloudera/sqoop/lib/FieldMappable.java 
> 2067ecc813146a41e7ccbcdd5472ffb29261db77 
>   src/java/com/cloudera/sqoop/lib/JdbcWritableBridge.java

[jira] [Commented] (SQOOP-3199) Sqoop 1.4.7 release preparation

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161130#comment-16161130
 ] 

Anna Szonyi commented on SQOOP-3199:


Hi [~maugli],

I know you've been very busy lately, so the delays are of course understandable 
with the release. 
Do you think you will be able to close off the release by the end of September, 
if not then we would be happy to take the shepherding of it over. Also, we 
would be happy to help with anything else (Bogi and Szabi ran the 3rd party 
tests), if needed.

There have been a few questions about the content as well, I will direct them 
to this umbrella jira.

Thanks,
Anna



> Sqoop 1.4.7 release preparation
> ---
>
> Key: SQOOP-3199
> URL: https://issues.apache.org/jira/browse/SQOOP-3199
> Project: Sqoop
>  Issue Type: Task
>Reporter: Attila Szabo
>Assignee: Attila Szabo
> Fix For: no-release
>
> Attachments: text.html
>
>
> Umbrella jira for 1.4.6 release.
> For reference, the release wikis are:
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release
> https://cwiki.apache.org/confluence/display/SQOOP/How+to+Release+Sqoop2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Add docker support to Sqoop test infrastructure to run third party tests automatically

[Anna Szonyi]

Hi Szabolcs,

The only thing I want to add/point out is that we will need to make sure to
reach out to apache legal regarding what we can add to the Sqoop repo and
into the Sqoop CI. We also need to confirm that the jenkins infra is ready
to run docker.

Other than this, this would be a very useful addition to our project,
thanks for driving this!

Thanks,
Anna

On Wed, Aug 30, 2017 at 11:40 AM, Szabolcs Vasas  wrote:

> Hi All,
>
> I have been working on making the third party test execution easier during
> the last couple of months and I would like to share my results/plans and
> ask for feedback.
> Executing the Sqoop third party test cases is way too painful and we have
> already discussed it in another chain that a possible solution to this
> would be to use docker containers to run the necessary RDBMSs. This
> solution could also enable us to run the third party test cases in our CI
> system which would be a really important step to improve the quality of
> Sqoop.
> I have created an umbrella JIRA for this initiative (
> https://issues.apache.org/jira/browse/SQOOP-3228) please feel free to take
> a look and leave a comment with your suggestions and ideas.
> My plan is to implement this initiative in 3 phases:
>
> Phase 1 (subtasks 1-5):
> The goal of this phase is to document how the contributors can setup the
> docker environment and start the necessary docker containers on their
> machines to be able to execute all the third party tests manually.
> This phase includes the necessary fixes in the test infrastructure and
> fixing the currently failing third party test cases too.
> We are actually very close to complete this phase I have managed to run the
> third party test cases with docker containers on my machine in 20 minutes,
> the documentation is the only step left.
>
> Phase 2 (subtasks 6-7):
> The goal of this phase is to automate the dockerized third party test
> execution. We want to have an ant task which starts up the necessary
> containers, executes the third party tests and shuts down the containers.
>
> Phase 3 (subtask 8):
> Change the Sqoop CI build to execute the third party test cases as well.
>
>
> Please let me know if you have any comments/concerns/suggestions regarding
> this initiative.
>
> Regards,
> Szabolcs
>

[jira] [Commented] (SQOOP-3196) Modify MySQLAuthTest to use configurable test database parameters

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143588#comment-16143588
 ] 

Anna Szonyi commented on SQOOP-3196:


Hi [~vasas],

Thanks for your contribution! 
Please feel free to resolve the jira and the review request.

Thanks,
Anna

> Modify MySQLAuthTest to use configurable test database parameters 
> --
>
> Key: SQOOP-3196
> URL: https://issues.apache.org/jira/browse/SQOOP-3196
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
> Attachments: SQOOP-3196.patch, SQOOP-3196.patch, SQOOP-3196.patch
>
>
> Most of the MySQL third party test cases use MySQLTestUtils to get the test 
> database parameters because these parameters can be configured using system 
> properties.
> One exception is MySQLAuthTest which uses a predefined database name, 
> username and password which cannot be configured. 
> The goal of this JIRA is to change MySQLAuthTest to use MySQLTestUtils since 
> it would make setting up a new test database easier and make the MySQL third 
> party test suite more consistent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3139) sqoop tries to re execute select query during import in case of a connection reset error and this is causing lots of duplicate records from source

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136985#comment-16136985
 ] 

Anna Szonyi commented on SQOOP-3139:


Hi [~Swank],

Thanks for the contribution, and congrats on your first commit! 
Please resolve the jira and set the review to "Submitted".

Thanks,
Anna

> sqoop tries to re execute select query during import in case of a connection 
> reset error and this is causing lots of duplicate records from source
> --
>
> Key: SQOOP-3139
> URL: https://issues.apache.org/jira/browse/SQOOP-3139
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: IBM Hadoop distribution: 4.2.0
> version: 1.4.6_IBM_27
> Sqoop 1.4.6_IBM_27
>Reporter: hemanth meka
>Assignee: Zoltán Tóth
> Fix For: 1.5.0
>
> Attachments: SQOOP-3139-3.patch, SQOOP-3139.patch
>
>
> We cannot reproduce this issue as it depends on network. Here is a edited log 
> content for understanding the issue. 
> Log start
> .
> .
> 2017-02-22 07:35:37,638 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:35:37,662 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:39:00,533 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>   at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>   at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>   at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>   at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>   at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>   at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>   at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>   at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>   at QueryResult.readFields0(QueryResult.java:10706)
>   at QueryResult.readFields(QueryResult.java:10415)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>   at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.ha

Re: Review Request 61814: SQOOP-2907 : SQOOP export from Hive table to RDBMS CLOB and BLOB CLOUMNS

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61814/#review183479
---



Hi Sandish,

Could you please add some test cases to this? Maybe on top of the existing 
HCatalogExportTests or a new test class.

Thanks,
Anna

- Anna Szonyi


On Aug. 22, 2017, 1:49 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61814/
> ---
> 
> (Updated Aug. 22, 2017, 1:49 p.m.)
> 
> 
> Review request for Sqoop and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3132
> https://issues.apache.org/jira/browse/SQOOP-3132
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP export from Hive table to RDBMS CLOB and BLOB CLOUMNS.
> export a String column from Hive table  to RDBS CLOB/BLOB data type column 
> using sqoop export.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/lib/JdbcWritableBridge.java afde5855 
>   src/java/org/apache/sqoop/lib/LobRef.java d6d6b253 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 
> 57a90b0a 
> 
> 
> Diff: https://reviews.apache.org/r/61814/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Re: Review Request 61777: sqoop tries to re execute select query during import in case of a connection reset error and this is causing lots of duplicate records from source

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61777/#review183322
---



Hi Zoltan,

Thanks for the contribution, and thanks for adding quite a few test cases, a 
few nitpicks above Szabolcs' ones.

Please take a look and let me know your thoughts.

Thanks,
Anna


src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java
Line 76 (original), 89-91 (patched)
<https://reviews.apache.org/r/61777/#comment259360>

This comment is a little hard to understand, could you give a bit more 
specific example?



src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java
Lines 93-99 (patched)
<https://reviews.apache.org/r/61777/#comment259361>

It might make sense to split the code into two separate methods for the 
"case sensitive match" and the "case insensitive match" cases.


- Anna Szonyi


On Aug. 21, 2017, 9:16 a.m., Zoltán Tóth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61777/
> ---
> 
> (Updated Aug. 21, 2017, 9:16 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3139
> https://issues.apache.org/jira/browse/SQOOP-3139
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> In the case if the database table name and split-by parameter was different 
> (eg.: Mycol vs. mycol) Sqoop couldn't continue the query from the last value 
> if the connection was broken.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java a78eb061 
>   src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java 
> 9a3621b0 
>   src/test/org/apache/sqoop/mapreduce/db/TestSQLServerDBRecordReader.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61777/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zoltán Tóth
> 
>

[jira] [Commented] (SQOOP-2411) Sqoop using '--direct' option fails with mysqldump exit code 2 and 3

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125598#comment-16125598
 ] 

Anna Szonyi commented on SQOOP-2411:


Hi [~sanysand...@gmail.com],

Thanks for following up on this jira!

In general we should only close these types of jiras if we know that we can't 
solve the issue from the Sqoop side/it's an expected failure/not a problem. 
However it might be a question around the cause of the exception: is the 
logging sufficient for the end user to tell what the root cause was, etc. Also 
it's a question of whether increasing 'net-write-timeout' or 'net-read-timeout' 
should solve these and if it's just a question of increasing it further (to how 
much) or if we're not passing it correctly (it's a bug on our end) or it 
doesn't have the desired effect (maybe a doc update).

In general if you could reproduce the issue, and think it's solvable, this 
could be an improvement to potentially improve logging or solve the time out 
issues/check whether the net-read-timeout increasing helps (or a doc jira about 
usage).

Thanks,
Anna

> Sqoop using '--direct' option fails with mysqldump exit code 2 and 3
> 
>
> Key: SQOOP-2411
> URL: https://issues.apache.org/jira/browse/SQOOP-2411
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/mysql
>Affects Versions: 1.4.6
> Environment: Amazon EMR
>Reporter: Karthick H
>Assignee: Sandish Kumar HN
>Priority: Critical
>
> I am running Sqoop in AWS EMR. I am trying to copy a table ~10 GB from MySQL 
> into HDFS.
> I get the following exception
> 15/07/06 12:19:07 INFO mapreduce.Job: Task Id : 
> attempt_1435664372091_0048_m_00_2, Status : FAILED
> Error: java.io.IOException: mysqldump terminated with status 3
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:485)
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
> 15/07/06 12:19:07 INFO mapreduce.Job: Task Id : 
> attempt_1435664372091_0048_m_05_2, Status : FAILED
> Error: java.io.IOException: mysqldump terminated with status 2
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:485)
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
> 15/07/06 12:19:08 INFO mapreduce.Job:  map 0% reduce 0%
> 15/07/06 12:19:20 INFO mapreduce.Job:  map 25% reduce 0%
> 15/07/06 12:19:22 INFO mapreduce.Job:  map 38% reduce 0%
> 15/07/06 12:19:23 INFO mapreduce.Job:  map 50% reduce 0%
> 15/07/06 12:19:24 INFO mapreduce.Job:  map 75% reduce 0%
> 15/07/06 12:19:25 INFO mapreduce.Job:  map 100% reduce 0%
> 15/07/06 12:23:11 INFO mapreduce.Job: Job job_1435664372091_0048 failed with 
> state FAILED due to: Task failed task_1435664372091_0048_m_00
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 15/07/06 12:23:11 INFO mapreduce.Job: Counters: 8
> Job Counters 
> Failed map tasks=28
> Launched map tasks=28
> Other local map tasks=28
> Total time spent by all maps in occupied slots (ms)=34760760
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=5793460
> Total vcore-seconds taken by all map tasks=5793460
> Total megabyte-seconds taken by all map tasks=8342582400
> 15/07/06 12:23:11 WARN mapreduce.Counters: Group FileSystemCounters is 
> deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
> 15/07/06 12:23:11 INFO mapreduce.ImportJobBase: Transferred

[jira] [Commented] (SQOOP-2907) Export parquet files to RDBMS: don't require .metadata for parquet files

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119600#comment-16119600
 ] 

Anna Szonyi commented on SQOOP-2907:


Hi [~yuan_zac],

Thanks for the original patch and the green light! :)

[~sanysand...@gmail.com],

Thank you for the great description of the contribution process! The only minor 
side note is the naming convention we have: it's 
SQOOP-(JIRANr)-(patchrevision).patch (meaning that if you have changed the 
patch 3 times due to a review it would be SQOOP-2907-3.patch).

Please go ahead and submit the rebased patch for review!

Thanks,
Anna

> Export parquet files to RDBMS: don't require .metadata for parquet files
> 
>
> Key: SQOOP-2907
> URL: https://issues.apache.org/jira/browse/SQOOP-2907
> Project: Sqoop
>  Issue Type: Improvement
>  Components: metastore
>Affects Versions: 1.4.6
> Environment: sqoop 1.4.6
> export parquet files to Oracle
>Reporter: Ruslan Dautkhanov
>Assignee: Sandish Kumar HN
> Attachments: SQOOP-2907.patch, SQOOP-2907.patch1
>
>
> Kite currently requires .metadata.
> Parquet files have their own metadata stored along data files.
> It would be great for Export operation on parquet files to RDBMS not to 
> require .metadata.
> We have most of the files created by Spark and Hive, and they don't create 
> .metadata, it only Kite that does.
> It makes sqoop export of parquet files usability very limited.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: New Sqoop Committer - Boglarka Egyed

[Anna Szonyi]

Congrats Bogi!! :)

On Tue, Aug 8, 2017 at 8:37 PM, Mark Kemper  wrote:

> Congrats Bogi
>
>
> Markus Kemper
> Customer Operations Engineer
> [image: www.cloudera.com] 
>
>
> On Tue, Aug 8, 2017 at 2:30 PM, Abraham Fine  wrote:
>
> > On behalf of the Apache Sqoop PMC, I am pleased to welcome Boglarka Eyed
> > as a new committer to Apache Sqoop. Please join me in congratulating her
> > for this accomplishment!
> >
> > Bogi has made a number of contributions to Sqoop particularly as it
> > relates to testing (especially Sqoop's 3rd party tests). In addition
> > Boglarka has performed many code reviews for the community. Boglarka's
> > reviews have been critical in our effort to maintain high quality code
> > and grow the community.
> >
> > Here is the JIRA listing of her contributions: https://s.apache.org/ruWN
> >
> > We are excited too see what Bogi contributes next!
> >
> > Thanks,
> > Abraham Fine
> >
>

[jira] [Comment Edited] (SQOOP-2907) Export parquet files to RDBMS: don't require .metadata for parquet files

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118505#comment-16118505
 ] 

Anna Szonyi edited comment on SQOOP-2907 at 8/8/17 3:53 PM:


Hi [~sanysand...@gmail.com],

Thanks for picking this up again! Please make sure that it applies to the 
current trunk (or if needed please rebase on it), also we should ping 
[~yuan_zac] and [~514793...@qq.com]] in case the original author wants to 
reclaim their patch. If not (or we receive no reply), then please submit it for 
review!

Thanks,
Anna


was (Author: anna.szonyi):
Hi [~sanysand...@gmail.com],

Thanks for picking this up again! Please make sure that it applies to the 
current trunk (or if needed please rebase on it), also we should ping 
[~yuan_zac] and [~chenkai.dr] in case the original author wants to reclaim 
their patch. If not (or we receive no reply), then please submit it for review!

Thanks,
Anna

> Export parquet files to RDBMS: don't require .metadata for parquet files
> 
>
> Key: SQOOP-2907
> URL: https://issues.apache.org/jira/browse/SQOOP-2907
> Project: Sqoop
>  Issue Type: Improvement
>  Components: metastore
>Affects Versions: 1.4.6
> Environment: sqoop 1.4.6
> export parquet files to Oracle
>Reporter: Ruslan Dautkhanov
>Assignee: Sandish Kumar HN
> Attachments: SQOOP-2907.patch, SQOOP-2907.patch1
>
>
> Kite currently requires .metadata.
> Parquet files have their own metadata stored along data files.
> It would be great for Export operation on parquet files to RDBMS not to 
> require .metadata.
> We have most of the files created by Spark and Hive, and they don't create 
> .metadata, it only Kite that does.
> It makes sqoop export of parquet files usability very limited.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (SQOOP-2907) Export parquet files to RDBMS: don't require .metadata for parquet files

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118505#comment-16118505
 ] 

Anna Szonyi edited comment on SQOOP-2907 at 8/8/17 3:52 PM:


Hi [~sanysand...@gmail.com],

Thanks for picking this up again! Please make sure that it applies to the 
current trunk (or if needed please rebase on it), also we should ping 
[~yuan_zac] and [~chenkai.dr] in case the original author wants to reclaim 
their patch. If not (or we receive no reply), then please submit it for review!

Thanks,
Anna


was (Author: anna.szonyi):
Hi [~sanysand...@gmail.com],

Thanks for picking this up again! Please make sure that it applies to the 
current trunk (or if needed please rebase on it), also we should ping 
[~yuan_zac] in case the original author wants to reclaim their patch. If not 
(or we receive no reply), then please submit it for review!

Thanks,
Anna

> Export parquet files to RDBMS: don't require .metadata for parquet files
> 
>
> Key: SQOOP-2907
> URL: https://issues.apache.org/jira/browse/SQOOP-2907
> Project: Sqoop
>  Issue Type: Improvement
>  Components: metastore
>Affects Versions: 1.4.6
> Environment: sqoop 1.4.6
> export parquet files to Oracle
>Reporter: Ruslan Dautkhanov
>Assignee: Sandish Kumar HN
> Attachments: SQOOP-2907.patch, SQOOP-2907.patch1
>
>
> Kite currently requires .metadata.
> Parquet files have their own metadata stored along data files.
> It would be great for Export operation on parquet files to RDBMS not to 
> require .metadata.
> We have most of the files created by Spark and Hive, and they don't create 
> .metadata, it only Kite that does.
> It makes sqoop export of parquet files usability very limited.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-2907) Export parquet files to RDBMS: don't require .metadata for parquet files

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118505#comment-16118505
 ] 

Anna Szonyi commented on SQOOP-2907:


Hi [~sanysand...@gmail.com],

Thanks for picking this up again! Please make sure that it applies to the 
current trunk (or if needed please rebase on it), also we should ping 
[~yuan_zac] in case the original author wants to reclaim their patch. If not 
(or we receive no reply), then please submit it for review!

Thanks,
Anna

> Export parquet files to RDBMS: don't require .metadata for parquet files
> 
>
> Key: SQOOP-2907
> URL: https://issues.apache.org/jira/browse/SQOOP-2907
> Project: Sqoop
>  Issue Type: Improvement
>  Components: metastore
>Affects Versions: 1.4.6
> Environment: sqoop 1.4.6
> export parquet files to Oracle
>Reporter: Ruslan Dautkhanov
>Assignee: Sandish Kumar HN
> Attachments: SQOOP-2907.patch, SQOOP-2907.patch1
>
>
> Kite currently requires .metadata.
> Parquet files have their own metadata stored along data files.
> It would be great for Export operation on parquet files to RDBMS not to 
> require .metadata.
> We have most of the files created by Spark and Hive, and they don't create 
> .metadata, it only Kite that does.
> It makes sqoop export of parquet files usability very limited.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3149) Sqoop incremental import - NULL column updates are not pulled into HBase table

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112700#comment-16112700
 ] 

Anna Szonyi commented on SQOOP-3149:


Hi [~jilani],

Thank you for your contribution and an extra thank you for being so patient 
throughout the entire process!

Thanks,
Anna

> Sqoop incremental import -  NULL column updates are not pulled into HBase 
> table
> ---
>
> Key: SQOOP-3149
> URL: https://issues.apache.org/jira/browse/SQOOP-3149
> Project: Sqoop
>  Issue Type: Bug
>  Components: connectors/generic, hbase-integration
>Affects Versions: 1.4.6
>Reporter: Jilani Shaik
> Attachments: hbase_delete_support_in_incremental_import
>
>
> Sqoop incremental import data from any database to HBase, if source table's 
> column from a row is updated to NULL, then target HBase table still showing 
> the previous value for that column. 
> So if you do a scan on the table for that row, HBase shows the previous 
> values of the column.
> Expected Result: Sqoop incremental import, If NULL columns are there in 
> source, then HBase need not store that and if it already exists need to 
> delete that column for a given row.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Changing behavior on trunk

[Anna Szonyi]

Hi All,

As no one voiced any concerns, I will commit the change tomorrow morning.

Thanks,
Anna

On Wed, Jul 26, 2017 at 2:36 PM, Boglarka Egyed <b...@cloudera.com> wrote:

> Hi All,
>
> +1 to Szabi's comment. As it is a bugfix I wouldn't think that a new option
> should be introduced to switch it on and off. However, it should be
> emphasized in the release notes (in whichever release it will be included)
> that this is a behavior changing fix.
>
> Thanks for bringing it up Anna!
>
> Cheers,
> Bogi
>
> On Fri, Jul 21, 2017 at 3:14 PM, Szabolcs Vasas <va...@cloudera.com>
> wrote:
>
> > Hi guys,
> >
> > I think that this is basically a bug fix and because of that I would not
> > add a new flag to switch it on/off.
> > It is true that it changes behavior but many other bug fixes do the same
> so
> > if we add a switch now we might want to do the same thing later and that
> > could lead to a large number of extra command line arguments.
> >
> > Regards,
> > Szabolcs
> >
> > On Thu, Jul 20, 2017 at 8:29 PM, Anna Szonyi <szo...@cloudera.com>
> wrote:
> >
> > > Hi @dev list,
> > >
> > > Jilani has created a great patch for SQOOP-3149
> > > <https://issues.apache.org/jira/browse/SQOOP-3149>: Sqoop incremental
> > > import - NULL column updates are not pulled into HBase table.
> > >
> > > This is a patch that (in my opinion) corrects previously incorrect
> > behavior
> > > in sqoop, however it can also be seen as just changing behavior from
> the
> > > users perspective ("we did not have nulls there before, now we have").
> > >
> > > The patch is ready to be committed, however I wanted to ask the dev
> list
> > on
> > > thoughts and opinions on whether we should merge the change as-is or
> > should
> > > we include a flag to turn this feature on and off? And if we need to
> > > include a flag, should it be in a separate commit or should we amend
> the
> > > original patch?
> > >
> > > Please find the review request below:
> > > https://reviews.apache.org/r/57499/
> > >
> > > Please chime in with your thoughts - if there are no objections on this
> > > thread, I plan to merge the change as-is in a week or so and pick up
> this
> > > conversation again after.
> > >
> > > Thanks,
> > > Anna
> > >
> >
> >
> >
> > --
> > Szabolcs Vasas
> > Software Engineer
> > <http://www.cloudera.com>
> >
>

[jira] [Comment Edited] (SQOOP-3178) SQOOP PARQUET INCREMENTAL MERGE

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096771#comment-16096771
 ] 

Anna Szonyi edited comment on SQOOP-3178 at 7/21/17 8:02 PM:
-

Hi [~sanysand...@gmail.com],

Thank you for your contribution, looking forward to future patches :)!

Please set the jira to resolved and the review request to submitted.

Thanks,
Anna


was (Author: anna.szonyi):
Hi [~sanysand...@gmail.com],

Thank you for your contribution, looking forward to future patches :)!

Thanks,
Anna

> SQOOP PARQUET INCREMENTAL MERGE 
> 
>
> Key: SQOOP-3178
> URL: https://issues.apache.org/jira/browse/SQOOP-3178
> Project: Sqoop
>  Issue Type: Improvement
>  Components: build, codegen, connectors
> Environment: None
>Reporter: Sandish Kumar HN
>Assignee: Sandish Kumar HN
>Priority: Blocker
>  Labels: features, newbie, sqoop
>
> Currently, sqoop-1 only supports merging of two Parquet format data sets but 
> it doesn't support to do incremental merge, so I have written a Sqoop 
> Incremental Merge MR for Parquet File Format and I have tested with million 
> records of data with N number of iterations.
> blocked by issue https://issues.apache.org/jira/browse/SQOOP-3192



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (SQOOP-3178) SQOOP PARQUET INCREMENTAL MERGE

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096771#comment-16096771
 ] 

Anna Szonyi edited comment on SQOOP-3178 at 7/21/17 8:01 PM:
-

Hi [~sanysand...@gmail.com],

Thank you for your contribution, looking forward to future patches :)!

Thanks,
Anna


was (Author: anna.szonyi):
Hi Sandish,

Thank you for your contribution, looking forward to future patches :)!

Thanks,
Anna

> SQOOP PARQUET INCREMENTAL MERGE 
> 
>
> Key: SQOOP-3178
> URL: https://issues.apache.org/jira/browse/SQOOP-3178
> Project: Sqoop
>  Issue Type: Improvement
>  Components: build, codegen, connectors
> Environment: None
>Reporter: Sandish Kumar HN
>Assignee: Sandish Kumar HN
>Priority: Blocker
>  Labels: features, newbie, sqoop
>
> Currently, sqoop-1 only supports merging of two Parquet format data sets but 
> it doesn't support to do incremental merge, so I have written a Sqoop 
> Incremental Merge MR for Parquet File Format and I have tested with million 
> records of data with N number of iterations.
> blocked by issue https://issues.apache.org/jira/browse/SQOOP-3192



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3178) SQOOP PARQUET INCREMENTAL MERGE

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096771#comment-16096771
 ] 

Anna Szonyi commented on SQOOP-3178:


Hi Sandish,

Thank you for your contribution, looking forward to future patches :)!

Thanks,
Anna

> SQOOP PARQUET INCREMENTAL MERGE 
> 
>
> Key: SQOOP-3178
> URL: https://issues.apache.org/jira/browse/SQOOP-3178
> Project: Sqoop
>  Issue Type: Improvement
>  Components: build, codegen, connectors
> Environment: None
>Reporter: Sandish Kumar HN
>Assignee: Sandish Kumar HN
>Priority: Blocker
>  Labels: features, newbie, sqoop
>
> Currently, sqoop-1 only supports merging of two Parquet format data sets but 
> it doesn't support to do incremental merge, so I have written a Sqoop 
> Incremental Merge MR for Parquet File Format and I have tested with million 
> records of data with N number of iterations.
> blocked by issue https://issues.apache.org/jira/browse/SQOOP-3192



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Changing behavior on trunk

[Anna Szonyi]

Hi @dev list,

Jilani has created a great patch for SQOOP-3149
: Sqoop incremental
import - NULL column updates are not pulled into HBase table.

This is a patch that (in my opinion) corrects previously incorrect behavior
in sqoop, however it can also be seen as just changing behavior from the
users perspective ("we did not have nulls there before, now we have").

The patch is ready to be committed, however I wanted to ask the dev list on
thoughts and opinions on whether we should merge the change as-is or should
we include a flag to turn this feature on and off? And if we need to
include a flag, should it be in a separate commit or should we amend the
original patch?

Please find the review request below:
https://reviews.apache.org/r/57499/

Please chime in with your thoughts - if there are no objections on this
thread, I plan to merge the change as-is in a week or so and pick up this
conversation again after.

Thanks,
Anna

Re: Review Request 59346: SQOOP-3178 : Incremental Merging for Parquet File Format

[Anna Szonyi]

> On July 20, 2017, 5:40 p.m., Anna Szonyi wrote:
> > Hi Sandish,
> > 
> > Thanks for all the changes, they aid readability a lot! I found a missing 
> > semicolon, other than that the tests seem to be running and everything is 
> > green.
> > 
> > Thanks,
> > Anna

Also please create the next patch with the name SQOOP-3178-13.patch.
Thanks,
Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59346/#review181060
---


On July 19, 2017, 4:49 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59346/
> ---
> 
> (Updated July 19, 2017, 4:49 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: PARQUET-1010 and SQOOP-3178
> https://issues.apache.org/jira/browse/PARQUET-1010
> https://issues.apache.org/jira/browse/SQOOP-3178
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> New feature for sqoop-1: Sqoop Merge and Incremental Merge for Parquet File 
> Format
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeJob.java 8b1cba33 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetReducer.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/ImportTool.java 78c7758e 
>   src/test/com/cloudera/sqoop/TestMerge.java 114e934a 
> 
> 
> Diff: https://reviews.apache.org/r/59346/diff/12/
> 
> 
> Testing
> ---
> 
> Hi,
> 
>  I have written a Sqoop Merge and Incremental Merge MR for Parquet File 
> Format and I have tested with million records of data with N number of 
> iterations. Please review My patch.
> 
> THIS ISSUE HAS DEPENDANCY ON PARQUET BUG. SO I HAVE RESOLVED THE PARQUET 
> ISSUE AND CREATED A PATCH FOR IT HERE 
> https://issues.apache.org/jira/browse/PARQUET-1010
> 
> Please let me know if there are any mistakes in My patch
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Re: Review Request 59346: SQOOP-3178 : Incremental Merging for Parquet File Format

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59346/#review181060
---



Hi Sandish,

Thanks for all the changes, they aid readability a lot! I found a missing 
semicolon, other than that the tests seem to be running and everything is green.

Thanks,
Anna


src/java/org/apache/sqoop/mapreduce/MergeJob.java
Lines 82 (patched)
<https://reviews.apache.org/r/59346/#comment256494>

this line is missing a ;


- Anna Szonyi


On July 19, 2017, 4:49 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59346/
> ---
> 
> (Updated July 19, 2017, 4:49 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: PARQUET-1010 and SQOOP-3178
> https://issues.apache.org/jira/browse/PARQUET-1010
> https://issues.apache.org/jira/browse/SQOOP-3178
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> New feature for sqoop-1: Sqoop Merge and Incremental Merge for Parquet File 
> Format
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeJob.java 8b1cba33 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetReducer.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/ImportTool.java 78c7758e 
>   src/test/com/cloudera/sqoop/TestMerge.java 114e934a 
> 
> 
> Diff: https://reviews.apache.org/r/59346/diff/12/
> 
> 
> Testing
> ---
> 
> Hi,
> 
>  I have written a Sqoop Merge and Incremental Merge MR for Parquet File 
> Format and I have tested with million records of data with N number of 
> iterations. Please review My patch.
> 
> THIS ISSUE HAS DEPENDANCY ON PARQUET BUG. SO I HAVE RESOLVED THE PARQUET 
> ISSUE AND CREATED A PATCH FOR IT HERE 
> https://issues.apache.org/jira/browse/PARQUET-1010
> 
> Please let me know if there are any mistakes in My patch
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Re: Review Request 59346: SQOOP-3178 : Incremental Merging for Parquet File Format

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59346/#review180878
---



Hi Sandish,

Thank you for making these changes so quickly! I have a few small comments, 
please address those, then I can +1 your patch.

Also for future reference our naming convention for patches is 
SQOOP-(issue#)-(revision#).patch, so if you have jira issue 123 and it's your 
4th revision in the review board, it should be SQOOP-123-4.patch :)

Also please attach the patch to the jira as well.

Thanks,
Anna


src/java/org/apache/sqoop/mapreduce/MergeJob.java
Lines 47 (patched)
<https://reviews.apache.org/r/59346/#comment256195>

apologies for nit picking, but unfortunately moving imports can cause a lot 
of merge conflicts later on, could you please reorder the lines as they were?



src/java/org/apache/sqoop/mapreduce/MergeJob.java
Line 128 (original)
<https://reviews.apache.org/r/59346/#comment256197>

another tiny nit-pick: it makes merging/backporting easier if we do not 
make unnecesary whitespace/new line changes.


- Anna Szonyi


On July 18, 2017, 8:53 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59346/
> ---
> 
> (Updated July 18, 2017, 8:53 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: PARQUET-1010 and SQOOP-3178
> https://issues.apache.org/jira/browse/PARQUET-1010
> https://issues.apache.org/jira/browse/SQOOP-3178
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> New feature for sqoop-1: Sqoop Merge and Incremental Merge for Parquet File 
> Format
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeJob.java 8b1cba33 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetReducer.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/ImportTool.java 78c7758e 
>   src/test/com/cloudera/sqoop/TestMerge.java 114e934a 
> 
> 
> Diff: https://reviews.apache.org/r/59346/diff/7/
> 
> 
> Testing
> ---
> 
> Hi,
> 
>  I have written a Sqoop Merge and Incremental Merge MR for Parquet File 
> Format and I have tested with million records of data with N number of 
> iterations. Please review My patch.
> 
> THIS ISSUE HAS DEPENDANCY ON PARQUET BUG. SO I HAVE RESOLVED THE PARQUET 
> ISSUE AND CREATED A PATCH FOR IT HERE 
> https://issues.apache.org/jira/browse/PARQUET-1010
> 
> Please let me know if there are any mistakes in My patch
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Re: Review Request 59346: SQOOP-3178 : Incremental Merging for Parquet File Format

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59346/#review180827
---



Hi Sandish,

Apologies for the delay in the review process, I went through the patch and it 
is a great contribution and will be very valuable to the users! 
I have a few review notes on it below, please check them out.

Thanks,
Anna


src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java
Lines 50 (patched)
<https://reviews.apache.org/r/59346/#comment256130>

this seems to be unused in the code.



src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java
Lines 53 (patched)
<https://reviews.apache.org/r/59346/#comment256129>

this seems to be an unused variable



src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java
Lines 58 (patched)
<https://reviews.apache.org/r/59346/#comment256131>

this seems to be unused.



src/java/org/apache/sqoop/mapreduce/MergeJob.java
Line 26 (original)
<https://reviews.apache.org/r/59346/#comment256111>

Thanks for removing these (I assume) unused imports, though it would make 
sense to remove all unused imports from this class in that case :)



src/java/org/apache/sqoop/mapreduce/MergeJob.java
Lines 54 (patched)
<https://reviews.apache.org/r/59346/#comment256109>

this seems to be an unused import.



src/java/org/apache/sqoop/mapreduce/MergeJob.java
Lines 207-208 (patched)
<https://reviews.apache.org/r/59346/#comment256126>

public static List readFooters(Configuration configuration, Path 
path) seems to be deprecated,
though it doesn't explicitly specify as far as I can tell 
public static List readFooters(Configuration configuration, 
FileStatus pathStatus, boolean skipRowGroups)  
this is the replacement method for it, it would make sense to use this 
instead.



src/java/org/apache/sqoop/mapreduce/MergeParquetMapper.java
Lines 55 (patched)
<https://reviews.apache.org/r/59346/#comment256127>

this seems to be an unused variable.



src/java/org/apache/sqoop/tool/ImportTool.java
Lines 495-499 (patched)
<https://reviews.apache.org/r/59346/#comment256106>

lastModifiedMergeParquet and lastModifiedMerge have a lot of similarities, 
it would make sense to refactor them, so we wouldn't have to repeat the code 
again.



src/test/com/cloudera/sqoop/TestMerge.java
Lines 57 (patched)
<https://reviews.apache.org/r/59346/#comment256110>

this seems to be an unused import.


- Anna Szonyi


On May 31, 2017, 8:25 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59346/
> ---
> 
> (Updated May 31, 2017, 8:25 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Szabolcs Vasas.
> 
> 
> Bugs: PARQUET-1010 and SQOOP-3178
> https://issues.apache.org/jira/browse/PARQUET-1010
> https://issues.apache.org/jira/browse/SQOOP-3178
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> New feature for sqoop-1: Sqoop Merge and Incremental Merge for Parquet File 
> Format
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/mapreduce/MergeGenericRecordExportMapper.java 
> PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeJob.java 8b1cba3 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/MergeParquetReducer.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/ImportTool.java 4b1b12d 
>   src/test/com/cloudera/sqoop/TestMerge.java 114e934 
> 
> 
> Diff: https://reviews.apache.org/r/59346/diff/5/
> 
> 
> Testing
> ---
> 
> Hi,
> 
>  I have written a Sqoop Merge and Incremental Merge MR for Parquet File 
> Format and I have tested with million records of data with N number of 
> iterations. Please review My patch.
> 
> THIS ISSUE HAS DEPENDANCY ON PARQUET BUG. SO I HAVE RESOLVED THE PARQUET 
> ISSUE AND CREATED A PATCH FOR IT HERE 
> https://issues.apache.org/jira/browse/PARQUET-1010
> 
> Please let me know if there are any mistakes in My patch
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Greetings and Reviews

[Anna Szonyi]

Hi @devs,

We've recently noticed that there is an increase in contributions to the
Sqoop project which has us super excited!
I would like to welcome all the new contributors to the project!

However due to this increased contribution activity we find ourselves short
on active reviewers. This causes some pile up on the patches and causes
delays for both the reviews and the commits.

We would like to ask our new contributors to please get involved with the
review process of each others' patches, as this will also hopefully reduce
the wait times.

We would also like to highlight that this is something that we will look at
when nominating contributors for committership :)

Thanks,
Anna

[jira] [Commented] (SQOOP-3197) Database credentials cannot be configured in PostgreSQL third party tests

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076627#comment-16076627
 ] 

Anna Szonyi commented on SQOOP-3197:


Hey Szabolcs,

Thanks for your continued improvements/standardization to the tests we are 
running!

Thanks,
Anna

> Database credentials cannot be configured in PostgreSQL third party tests
> -
>
> Key: SQOOP-3197
> URL: https://issues.apache.org/jira/browse/SQOOP-3197
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
> Attachments: SQOOP-3197.patch
>
>
> PostgreSQL third party tests look like they support configurable database 
> parameters (e.g. they load the database user, name and password from a system 
> property) but the property names in the test cases do not match the property 
> names in the build.xml and because of that these parameters cannot be 
> specified from the command line.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3206) Make sqoop fail if user uses --direct connector and tries to encode a null value when using a MySQL database

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076455#comment-16076455
 ] 

Anna Szonyi commented on SQOOP-3206:


Hey Zach,

Thank you for your great contribution! :)

Thanks,
Anna

> Make sqoop fail if user uses --direct connector and tries to encode a null 
> value when using a MySQL database
> 
>
> Key: SQOOP-3206
> URL: https://issues.apache.org/jira/browse/SQOOP-3206
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Zach Berkowitz
>Assignee: Zach Berkowitz
>Priority: Minor
> Attachments: sqoop-3206.patch
>
>
> The MySQL tools, mysqldump and mysqlimport, do not support custom null value 
> options, so sqoop should fail fast when --direct mode and 
> --input-null-string, --input-null-non-string, --null-string, or 
> --null-non-string are used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59844: Database credentials cannot be configured in PostgreSQL third party tests

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59844/#review179241
---



Hi Szabolcs,

Would you mind fixing the DirectPostgreSQLExportManualTest along with the other 
Postgres tests?

Thanks,
Anna

- Anna Szonyi


On June 6, 2017, 4:59 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59844/
> ---
> 
> (Updated June 6, 2017, 4:59 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3197
> https://issues.apache.org/jira/browse/SQOOP-3197
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> PostgreSQL third party tests look like they support configurable database 
> parameters (e.g. they load the database user, name and password from a system 
> property) but the property names in the test cases do not match the property 
> names in the build.xml and because of that these parameters cannot be 
> specified from the command line.
> 
> 
> Diffs
> -
> 
>   build.xml af43c47 
>   src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java 38d6ba6 
>   src/test/com/cloudera/sqoop/manager/PostgresqlExternalTableImportTest.java 
> 0515e09 
>   src/test/com/cloudera/sqoop/manager/PostgresqlImportTest.java 5872d25 
> 
> 
> Diff: https://reviews.apache.org/r/59844/diff/1/
> 
> 
> Testing
> ---
> 
> Ran unit and third party test cases.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 57499: Sqoop incremental import - NULL column updates are not pulled into HBase table

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57499/#review178977
---



Hi Jilani,

Thank you for your changes and apologies for the latency! I've reviewed the 
changes and they look really good! There is one last comment I am making that I 
want to ask your thoughts on and if we resolve that one I will commit your 
changes as soon as possible.

Thanks,
Anna


src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
Lines 172-200 (original), 174-217 (patched)
<https://reviews.apache.org/r/57499/#comment253359>

Hey Jilani,

Instead of this, maybe we could initialize the List at the top of the 
method and instead of using the booleans, like:

 List mutationList = new ArrayList<>();
  ...
for (...) {
 ...
Put put = new Put(Bytes.toBytes(rowKey));
if (null != val) {
  if ( val instanceof byte[]) {
put.addColumn(...);
  } else {
put.addColumn(...);
  }
  mutationList.add(put);
} else {
  Delete delete = new Delete(Bytes.toBytes(rowKey));
  delete.addColumn(...);
  mutationList.add(delete);
}
  }
}

return Collections.unmodifiableList(mutationList);
  }
  
or even use a Stack/Deque if you want to ensure that there is only a single 
(last) element that we return in either case:

Deque mutations = new ArrayDeque<>();
...
for (...) {
  ...
Put put = new Put(Bytes.toBytes(rowKey));
if (null != val) {
  if ( val instanceof byte[]) {
put.addColumn(...);
  } else {
put.addColumn(...);
  }
  mutations.addFirst(put);
} else {
  Delete delete = new Delete(Bytes.toBytes(rowKey));
  delete.addColumn(...)
  mutations.addFirst(delete);
}
  }
}

return Collections.singletonList(mutations.peekFirst());

Or something similar.
This way we don't have to track and check the isDelete and isPop, which 
makes the code a bit more clean and extensible.
Please let me know your thoughts on this!


- Anna Szonyi


On June 11, 2017, 8:21 p.m., Jilani Shaik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57499/
> ---
> 
> (Updated June 11, 2017, 8:21 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3149
> https://issues.apache.org/jira/browse/SQOOP-3149
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> HBase delete is added as part of incremental data import, Modified the return 
> type of method which is responsible for holding list of Put objects to List 
> of Mutation objects and at the time of execution of insert or delete using 
> the type of object in Mutation, executed the actaul operation either insert 
> or delete.
> 
> Jira ticket for the same: https://issues.apache.org/jira/browse/SQOOP-3149
> 
> Similar ticket to above:  SQOOP-3125
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/hbase/HBasePutProcessor.java fdbe1276 
>   src/java/org/apache/sqoop/hbase/PutTransformer.java 533467e5 
>   src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java 363e1456 
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java 58ccee7b 
>   src/test/com/cloudera/sqoop/hbase/HBaseImportTest.java fa14a013 
>   src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java a054eb66 
>   src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 6310a398 
> 
> 
> Diff: https://reviews.apache.org/r/57499/diff/6/
> 
> 
> Testing
> ---
> 
> Executed with jar prepared with these changes into hadoop cluster and tested 
> bot import and then incremental import.
> 
> 
> File Attachments
> 
> 
> HBase delete support for incremental import
>   
> https://reviews.apache.org/media/uploaded/files/2017/03/11/5b1895fd-4c6b-42fa-8a92-4e093153e370__hbase_delete_support_in_incremental_import
> updated with unit test and based on below suggestions
>   
> https://reviews.apache.org/media/uploaded/files/2017/03/22/708f63ba-2d8a-4a47-ab67-c1d2776354fd__hbase_delete_support_in_incremental_unit_test_also
> 
> 
> Thanks,
> 
> Jilani Shaik
> 
>

[jira] [Commented] (SQOOP-3194) HCatalogExportTest fails because of column escaping problems

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055514#comment-16055514
 ] 

Anna Szonyi commented on SQOOP-3194:


Thanks for the fix Szabolcs!

> HCatalogExportTest fails because of column escaping problems
> 
>
> Key: SQOOP-3194
> URL: https://issues.apache.org/jira/browse/SQOOP-3194
> Project: Sqoop
>  Issue Type: Test
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
> Attachments: SQOOP-3194.patch
>
>
> It seems there were some changes in the escaping logic in the test framework 
> which broke HCatalogExportTest. Since it is a third party test ant clean test 
> did not spot the problem earlier.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3198) Fix DirectMySQLExportTest and OracleExportTest

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055493#comment-16055493
 ] 

Anna Szonyi commented on SQOOP-3198:


Hey Szabi,

Thanks for catching and fixing this!

Thanks,
Anna

> Fix DirectMySQLExportTest and OracleExportTest
> --
>
> Key: SQOOP-3198
> URL: https://issues.apache.org/jira/browse/SQOOP-3198
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>  Labels: newbie
> Attachments: SQOOP-3198.patch
>
>
> DirectMySQLExportTest and OracleExportTest extend TestExport class and some 
> of the new test cases added in SQOOP-3158 fail in these tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SQOOP-3158) Columns added to Mysql after initial sqoop import, export back to table with same schema fails

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044116#comment-16044116
 ] 

Anna Szonyi commented on SQOOP-3158:


Hi Eric,

Another great contribution, thank you for your continued work to better Sqoop!

Thanks,
Anna

> Columns added to Mysql after initial sqoop import, export back to table with 
> same schema fails 
> ---
>
> Key: SQOOP-3158
> URL: https://issues.apache.org/jira/browse/SQOOP-3158
> Project: Sqoop
>  Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: viru reddy
>Assignee: Eric Lin
>  Labels: newbie
> Attachments: SQOOP-3158.2.patch, SQOOP-3158.3.patch, 
> SQOOP-3158.4.patch, SQOOP-3158.patch
>
>
> I have table in MySQL with 2 columns until yesterday. The columns are id and 
> name.
> 1,Raj
> 2,Jack
> I have imported this data into HDFS yesterday itself as a file. Today we 
> added a new column to the table in MySQL called salary. The table looks like 
> below.
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
> Now I have done Incremental import on this table as a file.
> Part-m-0 file contains
> 1,Raj
> 2,Jack
> Part-m-1 file contains
> 3,Jill,2000
> 4,Nick,3000
> Now I created a new table in MySQL with same schema as Original MySQL table 
> with columns id name and salary.
> When I do sqoop export only last 2 rows are getting inserted to the new table 
> in MySQL  and the sqoop export fails
> How can I reflect all the rows to be inserted to the table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58466: SQOOP-3158 - Columns added to Mysql after initial sqoop import, export back to table with same schema fails

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58466/#review177351
---


Ship it!




Ship It!

- Anna Szonyi


On May 30, 2017, 9:36 a.m., Eric Lin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58466/
> ---
> 
> (Updated May 30, 2017, 9:36 a.m.)
> 
> 
> Review request for Sqoop, Attila Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3158
> https://issues.apache.org/jira/browse/SQOOP-3158
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> I have table in MySQL with 2 columns until yesterday. The columns are id and 
> name.
> 1,Raj
> 2,Jack
> 
> I have imported this data into HDFS yesterday itself as a file. Today we 
> added a new column to the table in MySQL called salary. The table looks like 
> below.
> 
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
> 
> Now I have done Incremental import on this table as a file.
> 
> Part-m-0 file contains
> 1,Raj
> 2,Jack
> 
> Part-m-1 file contains
> 3,Jill,2000
> 4,Nick,3000
> 
> Now I created a new table in MySQL with same schema as Original MySQL table 
> with columns id name and salary.
> 
> Sqoop export will fail with below error:
> 
> java.lang.RuntimeException: Can't parse input data: 'Raj'
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:316)
> at SQOOP_3158.parse(SQOOP_3158.java:254)
> at 
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
> at 
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:854)
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:311)
> ... 12 more
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/ClassWriter.java eaa9123 
>   src/test/com/cloudera/sqoop/TestExport.java b2edc53 
>   src/test/org/apache/sqoop/TestExportUsingProcedure.java 519305c 
> 
> 
> Diff: https://reviews.apache.org/r/58466/diff/5/
> 
> 
> Testing
> ---
> 
> There is no existing test class to cover the path and I am not sure the best 
> way to add test case for this. If you have any suggestion, please let me know.
> 
> I have done manual testing to replicate the issue and confirmed that patch 
> fixed the issue. I have also tried different data types, all working.
> 
> However, if column in MySQL is defined as NOT NULL, then the export will 
> still fail with error, this is expected.
> 
> 
> Thanks,
> 
> Eric Lin
> 
>

Re: Review Request 59690: SQOOP-59690 : UPGRADE PARQUET TO NEW VERSION

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59690/#review177297
---


Ship it!




Ship It!

- Anna Szonyi


On May 31, 2017, 8:21 p.m., Sandish Kumar HN wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59690/
> ---
> 
> (Updated May 31, 2017, 8:21 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Brock Noland, Attila Szabo, and 
> Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3192
> https://issues.apache.org/jira/browse/SQOOP-3192
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> UPGRADE PARQUET TO NEW VERSION
> 
> 
> Diffs
> -
> 
>   ivy/libraries.properties 84b834b 
> 
> 
> Diff: https://reviews.apache.org/r/59690/diff/1/
> 
> 
> Testing
> ---
> 
> UPGRADE PARQUET TO NEW VERSION. 
> 
> SQOOP is using parquet through KITESDK so I have update 
> kite-data.version=1.0.0-> kite-data.version=1.1.0 which is the new version.
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>

Re: Review Request 57499: Sqoop incremental import - NULL column updates are not pulled into HBase table

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57499/#review173716
---



Hi,

Thank you for this patch, this seems like a very useful change! There are a few 
suggestions/questions I've added, please look through them and fix/comment as 
needed :). Also thank you for creating a test case for this, that was a lot of 
help with the review and testing!

Thanks,
Anna


src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
Line 191 (original), 196 (patched)
<https://reviews.apache.org/r/57499/#comment250697>

it seems Put class' add method was deprecated in hbase 1.0.0 and we're 
currently using hbase 1.2.4, please use put.addColumn instead.



src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
Lines 205 (patched)
<https://reviews.apache.org/r/57499/#comment250696>

it seems deleteColumn was deprecated in hbase 1.0.0 and we're currently 
using hbase 1.2.4, please use delete.addColumn instead.



src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
Lines 211 (patched)
<https://reviews.apache.org/r/57499/#comment246762>

It might be cleaner if you initialize the list of mutations sooner and add 
the correct put or delete to the list based on the val == null instead of 
carrying the booleans isPut and isDelete (or create a mutation object and carry 
that around, in that case you can create an immutable list instead of a regular 
ArrayList).



src/test/com/cloudera/sqoop/hbase/HBaseImportTest.java
Lines 85 (patched)
<https://reviews.apache.org/r/57499/#comment250725>

Currently we're only testing the null update for lastmodified. Based on my 
understanding this should not affect append mode, could you please add a test 
case to verify that this change does not affect append mode, if you agree with 
my assumption.



src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
Line 118 (original), 125-156 (patched)
<https://reviews.apache.org/r/57499/#comment250724>

Quite a bit of this function is already in the original getArgv function in 
the HBaseTestCase class, please reuse/extract-refactor that part of the method.


- Anna Szonyi


On April 9, 2017, 5:49 p.m., Jilani Shaik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57499/
> ---
> 
> (Updated April 9, 2017, 5:49 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, Attila Szabo, and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3149
> https://issues.apache.org/jira/browse/SQOOP-3149
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> HBase delete is added as part of incremental data import, Modified the return 
> type of method which is responsible for holding list of Put objects to List 
> of Mutation objects and at the time of execution of insert or delete using 
> the type of object in Mutation, executed the actaul operation either insert 
> or delete.
> 
> Jira ticket for the same: https://issues.apache.org/jira/browse/SQOOP-3149
> 
> Similar ticket to above:  SQOOP-3125
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/hbase/HBasePutProcessor.java fdbe1276 
>   src/java/org/apache/sqoop/hbase/PutTransformer.java 533467e5 
>   src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java 363e1456 
>   src/java/org/apache/sqoop/mapreduce/HBaseBulkImportMapper.java 58ccee7b 
>   src/test/com/cloudera/sqoop/hbase/HBaseImportTest.java fa14a013 
>   src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java a054eb66 
>   src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 6310a398 
> 
> 
> Diff: https://reviews.apache.org/r/57499/diff/4/
> 
> 
> Testing
> ---
> 
> Executed with jar prepared with these changes into hadoop cluster and tested 
> bot import and then incremental import.
> 
> 
> File Attachments
> 
> 
> HBase delete support for incremental import
>   
> https://reviews.apache.org/media/uploaded/files/2017/03/11/5b1895fd-4c6b-42fa-8a92-4e093153e370__hbase_delete_support_in_incremental_import
> updated with unit test and based on below suggestions
>   
> https://reviews.apache.org/media/uploaded/files/2017/03/22/708f63ba-2d8a-4a47-ab67-c1d2776354fd__hbase_delete_support_in_incremental_unit_test_also
> 
> 
> Thanks,
> 
> Jilani Shaik
> 
>

Re: SQOOP 1.4.7 Release by end of June

[Anna Szonyi]

Hi Attila,

This would be really great! Please let me know if I can help with anything!

Thanks,
Anna

On Thu, May 25, 2017 at 6:20 PM, Attila Szabó  wrote:

> Hello everyone,
>
> In the past few months we've mentioned it a few times that it would make
> sense to create a new release of Sqoop from the current trunk version
> (1.4.7).
>
> IMHO this is the right time to move forward with this topic.
> First of all because the latest release is 2 years old (1.4.6 was released
> in 2015 May), and it would make a great impact for the end users to deliver
> all of the changes we as a community have made since.
> As a second argument, recently we were able to onboard quite a few new
> contributor (including but not limited to Illya Yalovyy, Ying Cao,  Eric
> Lin, Chris Teoh, Dmitry Zagorulkin), and as a recognition of their work,
> I'd like to ensure their commits gets into a release as soon as possible.
> Third: recently Anna, Bogi and Szabi were super active on the front of
> stabilizing and improving the tests, the build system and so, however
> according to my feelings the current status quo (waiting for a minor
> version change with bigger changes) holding them back from more fundamental
> improvements (I guess this is the reason why the Gradle/Maven new build
> sytem is still in progress, and the conversation around feature branches
> started). I think it would make a great impact for the whole community and
> the product itself, if we would be able to remove this burden from their
> shoulders, and thus they would be able to focus on bigger
> changes/challenges.
>
> Because of the reasons mentioned above, on the top of my existing duties,
> I'd like to help the community's life by owning the 1.4.7 release process,
> and starting the release with branching the trunk as of 2017-06-01 (next
> Friday) PST 8PM, and finishing the whole process latest by the end of June.
>
> Of course before starting even creating the related JIRA tasks, plans, the
> branching, etc. I'd like to collect the inputs from all of you if this fits
> your plan/requirements/etc.
>
> Looking forward hearing all of your thoughts, concerns, pros and cons.
>
> Best regards,
> Attila Szabo
>

[jira] [Commented] (SQOOP-816) Scoop and support for external Hive tables

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012005#comment-16012005
 ] 

Anna Szonyi commented on SQOOP-816:
---

Hi [~chris.t...@gmail.com],

Thank you for your contribution!

Thanks,
Anna

> Scoop and support for external Hive tables
> --
>
> Key: SQOOP-816
> URL: https://issues.apache.org/jira/browse/SQOOP-816
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hive-integration
>Reporter: Santosh Achhra
>Assignee: Chris Teoh
>Priority: Minor
>  Labels: External, Hive,, Scoop,, Tables, newbie
>
> Sqoop is not supporting HIVE external tables at the moment. Any imports using 
> scoop creates a managed table, in real world scenario it is very important to 
> have EXTERNAL tables. As of now we have to execute ALTER statement to change 
> table properties to make the the table as external table which is not a big 
> deal but it would nice have an option in scoop to specify type of table which 
> is required



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 51912: SQOOP 816 - Scoop and support for external Hive tables

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51912/#review174891
---



Hi Chris,

Thanks for the NPE fixes! Unfortunately I don't see the test class 
src/test/org/apache/sqoop/hive/TestTableDefWriter.java in the current revision 
(7). Could you please add that back in? Also it generally makes it easier if 
you name the patches including the revisions (SQOOP-816-7.patch), as it's 
easier to track what you downloaded and applied and what changed between the 
versions.

As a side note (again this is not a must), but do you think you could include 
the Postgres E2E test you were running (maybe extending HiveImportTest or the 
PostgresqlImportTests).

Apologies for the latency on this, this is a great, super useful feature and we 
want to make sure it gets committed with all the necessary classes/features!

Thanks,
Anna

- Anna Szonyi


On April 30, 2017, 7:48 a.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51912/
> ---
> 
> (Updated April 30, 2017, 7:48 a.m.)
> 
> 
> Review request for Sqoop, Kathleen Ting, Attila Szabo, and Liz Szilagyi.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-816 - Hive External table support. Added --as-external-table option and 
> --table-location option
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/SqoopOptions.java 801942e 
>   src/java/org/apache/sqoop/hive/TableDefWriter.java 32fcca3 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 46f405f 
>   src/test/org/apache/sqoop/tool/TestImportTool.java 7e11f54 
> 
> 
> Diff: https://reviews.apache.org/r/51912/diff/7/
> 
> 
> Testing
> ---
> 
> Unit tests.
> Interactive IDE debug step throughs.
> No testing on HiveServer as SQOOP-2787 may not yet be in trunk.
> End to end testing on PostgresQL to HiveServer performed.
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Creating feature branches

[Anna Szonyi]

Hi @devs,

We would like to start the preparation for upgrading to hive 2 (upgrade
version, migrate off of hive cli and onto beeline, etc.).
As we anticipate there being a few patches we were thinking it would make
sense to create a feature branch (sqoop_hive2) for this, as we do not want
trunk to become unstable or introduce "breaking changes" before the
proposed 1.4.7 release.

Does anyone have any concerns about this?

And in general do we have a policy about feature branches for sqoop (like a
sqoop_kerberos branch, etc.)?

Thanks,
Anna

Re: Review Request 51912: SQOOP 816 - Scoop and support for external Hive tables

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51912/#review172789
---



Hi Chris,

Thank you for yout contribution, thisis a great change! There are a few places 
I found that can -and probably will- cause NullPointerExceptions, please fix 
these, as they're currently causing some tests to fail as well.
It might be out of scope for this review, but it would be great if you could 
add new tests for to the TestHiveImport test class which can cover an external 
hive table import end to end (it could even be a parameterized test, so all 
tests would run once for external and once for non-external :)). 

Otherwise the change looks pretty nice and clear, thanks! :)

Thanks,
Anna


src/java/org/apache/sqoop/hive/TableDefWriter.java
Lines 136 (patched)
<https://reviews.apache.org/r/51912/#comment245834>

This line has a potential NPE in it (currently some tests run with ant test 
fail with NPE on one of these lines).
Replacing it with StringUtils.isBlank() would help with this.
It might also make sense to extract the whole condition like 
isHiveExternalTableSet or something similar



src/java/org/apache/sqoop/hive/TableDefWriter.java
Lines 142 (patched)
<https://reviews.apache.org/r/51912/#comment245835>

This line has a potential NPE.
Replacing it with StringUtils.isBlank()  would help with this.
It might also make sense to extract the whole condition like 
isHiveExternalTableSet or something similar.



src/java/org/apache/sqoop/hive/TableDefWriter.java
Lines 231 (patched)
<https://reviews.apache.org/r/51912/#comment245836>

This line has a potential NPE in it.
Replacing it with StringUtils.isBlank()  would help with this.
It might also make sense to extract the whole condition like 
isHiveExternalTableSet or something similar



src/java/org/apache/sqoop/tool/BaseSqoopTool.java
Lines 1583 (patched)
<https://reviews.apache.org/r/51912/#comment245833>

This line has a potential NPE.
Replacing it with StringUtils.isBlank()  would help with this.
It might also make sense to extract the whole condition to make it a bit 
more easy to understand, e.g. isNotHiveImportButExternalTableDirIsSet or 
something similar :).



src/test/org/apache/sqoop/hive/TestTableDefWriter.java
Lines 82 (patched)
<https://reviews.apache.org/r/51912/#comment245842>

Thanks for creating a new test class! Would you please also add test cases 
that cover the "default path", where we're not creating external tables, to 
ensure that is still working as expected?


- Anna Szonyi


On April 21, 2017, 1:45 p.m., Chris Teoh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51912/
> ---
> 
> (Updated April 21, 2017, 1:45 p.m.)
> 
> 
> Review request for Sqoop, Kathleen Ting, Attila Szabo, and Liz Szilagyi.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> SQOOP-816 - Hive External table support. Added --as-external-table option and 
> --table-location option
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/SqoopOptions.java 801942e 
>   src/java/org/apache/sqoop/hive/TableDefWriter.java 32fcca3 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 46f405f 
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java PRE-CREATION 
>   src/test/org/apache/sqoop/tool/TestImportTool.java 7e11f54 
> 
> 
> Diff: https://reviews.apache.org/r/51912/diff/5/
> 
> 
> Testing
> ---
> 
> Unit tests.
> Interactive IDE debug step throughs.
> No testing on HiveServer as SQOOP-2787 may not yet be in trunk.
> End to end testing on PostgresQL to HiveServer performed.
> 
> 
> Thanks,
> 
> Chris Teoh
> 
>

Re: Review Request 58295: SQOOP-3169: Evaluate and fix SQLServer Manual tests

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58295/#review171576
---


Ship it!




Ship It!

- Anna Szonyi


On April 10, 2017, 12:47 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58295/
> ---
> 
> (Updated April 10, 2017, 12:47 p.m.)
> 
> 
> Review request for Sqoop and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3169
> https://issues.apache.org/jira/browse/SQOOP-3169
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Fixed and improved SQLServer Manual tests so that they can be executed at 
> once by setting DB connect string anf credentials via system properties. This 
> patch also contains some more improvement, like:
> - ignored invalid test cases
> - added database cleanup logic in tearDown part
> - updated java docs
> - removed unused imports
> 
> 
> Diffs
> -
> 
>   build.xml 73db28b272c50b4f76fef8421e6b9dfe5fed40f4 
>   src/test/com/cloudera/sqoop/hive/TestHiveImport.java 
> 33e0cc41f6f379bac2085431e0f1adc60bce6bce 
>   src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java 
> 9a92479245fa35c210d8e49f847292ee53d6f9b1 
>   src/test/com/cloudera/sqoop/manager/SQLServerManagerImportManualTest.java 
> 1f69725da8408853ac55b1f316ce1b9ef015e674 
>   src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java 
> 851bf49614e829d07de252b83f4ad550d0cb043b 
>   src/test/org/apache/sqoop/manager/sqlserver/ManagerCompatExport.java 
> 8c5176ad61aae61b96c7458d3b4b83dc11960268 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeExportDelimitedFileManualTest.java
>  099d7344beb428c58b32d926af5ea079211da490 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeExportSequenceFileManualTest.java
>  21676f02510693dcdd856a1d9dfba7d05eace023 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeImportDelimitedFileManualTest.java
>  519fb525bdbb167520368d404667036669925041 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeImportSequenceFileManualTest.java
>  a0dad8a60b99d522ad3691e15b8b16c56e4b5858 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportManualTest.java
>  1999272181421a539318ed195ea4257f52b2ed08 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerManualTest.java 
> 1178e3c79de4d0b5c7a96c6ad7eb316ed15e47c4 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerMultiColsManualTest.java 
> 6a8ab51967237f471044b868615fdb3e057b1d92 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerMultiMapsManualTest.java 
> c9a5b5ef596cfc1b28948c2a071935dfb9500cde 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerParseMethodsManualTest.java
>  cd05aecf1ae5bd79fb485325d58b33a73e9df290 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerQueryManualTest.java 
> 0057ac9df562c8e92cf7b9014c5e4239886a8104 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerSplitByManualTest.java 
> f85245ab8cdd66da983ac9017d356f251f22e7db 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerWhereManualTest.java 
> 10ae03b324b15f5ea0cc3cbbc04d3a5041233dd9 
> 
> 
> Diff: https://reviews.apache.org/r/58295/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test
> 
> ant clean test -Dsqoop.thirdparty.lib.dir=3rdparty 
> -Dsqoop.test.sqlserver.connectstring.host_url=sqlserver_connectstring 
> -Dsqoop.test.sqlserver.database=databasename -Dms.sqlserver.username=username 
> -Dms.sqlserver.password=password -Dtestcase=SQLServer*
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Running 3rd party test on the CI

[Anna Szonyi]

Hi All,

We started a discussion about running thirdparty integration tests in our
CI on separate threads (another e-mail chain and a review), and I thought
it might make sense to bring these threads together.

We were discussing the best way of running the thirdparty tests on the
apache CI, as we think it would be very beneficial.

We were floating around the idea of trying to get resources (mysql, cubrid,
etc.) "donated", if that's possible (as far as I understand that's more or
less how Kudu deals with the same issue).

Attila proposed to investigate using docker images.

I would like to bring this up on the dev list - in case anyone has any
experience with this (using external resources from apache CIs), or anyone
has any concerns/negative experiences.

Please chime in here if you have any ideas/workable solutions for this or
any objections.

Thanks,
Anna

Re: Sqoop CI changes

[Anna Szonyi]

Hey Attila,

Thanks for your input! I agree that adding the 3rd party tests to the CI
would be really beneficial. As there are some resourcing problems that need
to be solved for that to happen (I started a discussion with the Kudu team,
however if you have any specific help to add there, I would be very happy
to accept any assistance with that) - I'll start a separate thread where we
can discuss this issue with the community, see if anyone else has any
inputs on it :).

However if no one has any objections about the original proposal, I will
follow through with that in the meantime.

Thanks and Regards,
Anna


On Fri, Apr 7, 2017 at 1:04 PM, Attila Szabó <mau...@apache.org> wrote:

> Hello everyone,
>
> First of all I'd like to thank that Anna is willing to invest some efforts
> on making our CI system better, and sorry for my delayed answer ( however I
> do still hope it helps regardless the stage of the ongoing efforts )
>
> I'd like to share the following thoughts here:
> - Although it would make sense to eliminate the four ( right now totally
> equal ) CI cycles and create only one, but regardless some "static noise"
> it doesn't cause any serious issues for our current commit flow.
> - Creating a precommit hook sounds like a great idea, I would encourage the
> community to move on that path.
> - However: According to my humblest opinion the biggest problem with the
> current CI is that it doesn't execute the so called " 3rd party tests" (
> which is generally our DB integration test layer), and thus it provides
> only a limited safety belt for us ( and we've seen quite a few regression
> on this front in the past one year ). Although we do have solution for
> running those tests manually from command line, it's quite difficult to
> setup/test those things from a single desktop, thus cause serious
> difficulties in validating some changeset before commit.
> - Still an issue we have on this front is our build
> system/scripts/mechanism, which again could slow down the Dev+commit flow.
> Although we've started efforts on this front, the final solution was not
> fully delivered yet.
>
> As a conclusion of the elements above:
> Anna! Would you mind first focusing on the build scripts and the "3rd
> party" CI automation instead of eliminating the obsolete stuff? IMHO that
> would be a much better usage of your efforts and would provide a much
> bigger impact for the community.
>
> With my kindest regards,
> Attila
>
> On Mar 28, 2017 2:44 PM, "Erzsebet Szilagyi" <liz.szila...@cloudera.com>
> wrote:
>
> > Great ideas!
> >
> > I agree with Bogi and Szabolcs on the redundant test jobs.
> >
> > Would this pre-commit hook launch the same process as the current
> > post-commit hook, or would this do something different?
> > I think in the first case we could rework the post-commit check into the
> > pre-commit hook, in the latter I'm curious about what exactly this check
> > would add.
> > In general I support the idea: we have seen a number of problems that
> could
> > have been avoided, so this shall be a very useful change!
> >
> > Thank you,
> > Liz
> >
> > On Fri, Mar 24, 2017 at 9:39 AM, Szabolcs Vasas <va...@cloudera.com>
> > wrote:
> >
> > > Hi Anna,
> > >
> > > Removing the redundant test execution jobs sounds great, I think you
> can
> > go
> > > ahead with that.
> > >
> > > Regarding the pre-commit hook: what would be the purpose of it exactly?
> > > Would it execute the unit tests before the patch is committed?
> > >
> > > Regards,
> > > Szabolcs
> > >
> > > On Thu, Mar 23, 2017 at 4:03 PM, Anna Szonyi <szo...@cloudera.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to make the following changes to the Sqoop CI system:
> > > > Disable the SCM polling for the Sqoop-hadoop23 Sqoop-hadoop20 and
> > > > Sqoop-hadoop100 jobs (and later delete the jobs themselves),
> > > > as the current trunk version of sqoop no longer contains these
> > profiles,
> > > so
> > > > these runs are redundant.
> > > >
> > > > I would also like to propose the creation of a pre-commit hook for
> > Sqoop
> > > > (like the existing one for Sqoop2).
> > > >
> > > > Please let me know if you have any objections.
> > > >
> > > > Thanks,
> > > > Anna
> > > >
> > >
> > >
> > >
> > > --
> > > Szabolcs Vasas
> > > Software Engineer
> > > <http://www.cloudera.com>
> > >
> >
>

[jira] [Commented] (SQOOP-3168) Sqoop Saved Job feature of overwriting job argument at execution time is not working in Sqoop1.4.6CDH 5.8.0

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960653#comment-15960653
 ] 

Anna Szonyi commented on SQOOP-3168:


Hi,

Unfortunately this is a bug that was introduced with SQOOP-2779 and later fixed 
by SQOOP-2896. As for a workaround: there is none, besides patching it with the 
fix of SQOOP-2896, as for CDH 5.8.0, only creating new jobs or upgrading to 
5.8.5+ would solve it, as that is where the fix was first backported.

Apologies and Regards,
Anna

> Sqoop Saved Job feature of overwriting job argument at execution time is not 
> working in Sqoop1.4.6CDH 5.8.0
> ---
>
> Key: SQOOP-3168
> URL: https://issues.apache.org/jira/browse/SQOOP-3168
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Hemendra Yadav
>
> Hi,
> I have created one sqoop saved job using below command ::
> sqoop job --create  -- import --connect  --driver 
> com.mysql.jdbc.Driver --username  --password  --table 
>  --target-dir location1 --fields-terminated-by , --escaped-by \\ -m 1
>  
> I need to put the data to different location so while execution i have 
> provided new location in below command::
> sqoop job --exec jobname1 -- --target-dir location_new
>  But it still picks up the target directory from the saved job i.e. location1.
> I am currently using sqoop 1.4.6 CDH 5.8.0 version. When i have checked the 
> sqoop code it seems that in sqoop CDH 5.8.0  code for overwritting job 
> argument is removed from the method "private int execJob(SqoopOptions opts)" 
> in the class "org.apache.sqoop.tool.JobTool" due to issue 
> "https://issues.apache.org/jira/browse/SQOOP-2779;.
>  When i have executed the same command using sqoop 1.4.6 CDH 5.7.3 it is able 
> to overwrite the target directory and working as expected.
> Can you please suggest any workaround how to import data to a different 
> target directory while executing sqoop saved job.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58233: SQOOP-3167: Evalutation and automation of SQLServer Manual tests

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58233/#review171209
---



Hey Bogi,

Thanks for looking through, cleaning up and adding the SQLServer tests to the 
other thirdparty tests, great work!

Please see my two comments below. 

Thanks,
Anna


src/test/com/cloudera/sqoop/hive/TestHiveImport.java
Line 385 (original), 385 (patched)
<https://reviews.apache.org/r/58233/#comment244090>

While you didn't name this method, do you think it would make sense to 
rename it to getTypes or something similar?



src/test/com/cloudera/sqoop/hive/TestHiveImport.java
Line 463 (original), 463 (patched)
<https://reviews.apache.org/r/58233/#comment244089>

Why was the AsParquet removed from the method name?


- Anna Szonyi


On April 6, 2017, 2:21 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58233/
> ---
> 
> (Updated April 6, 2017, 2:21 p.m.)
> 
> 
> Review request for Sqoop, Anna Szonyi, Szabolcs Vasas, and Liz Szilagyi.
> 
> 
> Bugs: SQOOP-3167
> https://issues.apache.org/jira/browse/SQOOP-3167
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Automated SQLServer Manual tests including test case correction and minor 
> rework too:
> - modified connection string setup to make them able to run automatically
> - fixed failing test cases
> - excluded invalid test cases
> - added database cleanup logic in tearDown part
> - updated java docs
> - removed unused imports
> - changing names to add them to the 3rd party test suite
> 
> Note: A more extensive refactor of the test classes in 
> org.apache.sqoop.manager.sqlserver could be made, it will be addressed in 
> another JIRA as that is a different scope.
> 
> 
> Diffs
> -
> 
>   build.xml 73db28b272c50b4f76fef8421e6b9dfe5fed40f4 
>   src/test/com/cloudera/sqoop/hive/TestHiveImport.java 
> 33e0cc41f6f379bac2085431e0f1adc60bce6bce 
>   src/test/com/cloudera/sqoop/manager/SQLServerManagerExportManualTest.java 
> 9a92479245fa35c210d8e49f847292ee53d6f9b1 
>   src/test/com/cloudera/sqoop/manager/SQLServerManagerImportManualTest.java 
> 1f69725da8408853ac55b1f316ce1b9ef015e674 
>   src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java 
> 851bf49614e829d07de252b83f4ad550d0cb043b 
>   src/test/org/apache/sqoop/manager/sqlserver/ManagerCompatExport.java 
> 8c5176ad61aae61b96c7458d3b4b83dc11960268 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeExportDelimitedFileManualTest.java
>  099d7344beb428c58b32d926af5ea079211da490 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeExportSequenceFileManualTest.java
>  21676f02510693dcdd856a1d9dfba7d05eace023 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeImportDelimitedFileManualTest.java
>  519fb525bdbb167520368d404667036669925041 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerDatatypeImportSequenceFileManualTest.java
>  a0dad8a60b99d522ad3691e15b8b16c56e4b5858 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportManualTest.java
>  1999272181421a539318ed195ea4257f52b2ed08 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerManualTest.java 
> 1178e3c79de4d0b5c7a96c6ad7eb316ed15e47c4 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerMultiColsManualTest.java 
> 6a8ab51967237f471044b868615fdb3e057b1d92 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerMultiMapsManualTest.java 
> c9a5b5ef596cfc1b28948c2a071935dfb9500cde 
>   
> src/test/org/apache/sqoop/manager/sqlserver/SQLServerParseMethodsManualTest.java
>  cd05aecf1ae5bd79fb485325d58b33a73e9df290 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerQueryManualTest.java 
> 0057ac9df562c8e92cf7b9014c5e4239886a8104 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerSplitByManualTest.java 
> f85245ab8cdd66da983ac9017d356f251f22e7db 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerWhereManualTest.java 
> 10ae03b324b15f5ea0cc3cbbc04d3a5041233dd9 
> 
> 
> Diff: https://reviews.apache.org/r/58233/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test, ant test
> 
> ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=3rdpartylib 
> -Dsqoop.test.sqlserver.connectstring.host_url=sqlserver 
> -Dsqoop.test.sqlserver.database=databasename -Dms.sqlserver.username=username 
> -Dms.sqlserver.password=password -Dtestcase=SQLS

[jira] [Commented] (SQOOP-3159) Sqoop (export + --table) with Oracle table_name having '$' fails with error (ORA-00942 or java.lang.NoClassDefFoundError)

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943364#comment-15943364
 ] 

Anna Szonyi commented on SQOOP-3159:


Thank you for your contribution Szabolcs!

Thanks,
Anna

> Sqoop (export + --table) with Oracle table_name having '$' fails with error 
> (ORA-00942 or java.lang.NoClassDefFoundError)
> -
>
> Key: SQOOP-3159
> URL: https://issues.apache.org/jira/browse/SQOOP-3159
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Markus Kemper
>Assignee: Szabolcs Vasas
> Attachments: SQOOP-3159.patch
>
>
> Sqoop (export + --table) with Oracle table_name having '$' fails with error 
> (ORA-00942 or java.lang.NoClassDefFoundError), see test case below.
> *Test Case*
> {noformat}
> #
> # STEP 01 - Create Data
> #
> export MYCONN=jdbc:oracle:thin:@oracle1.cloudera.com:1521/db11g;
> export MYUSER=sqoop
> export MYPSWD=cloudera
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "create table t1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1"
> Output:
> ---
> | C1   | C2  | C3 | 
> ---
> | 1| 2017-03-21 13:30:02.0 | some data  | 
> ---
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "drop table t\$1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "create table t\$1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "insert into t\$1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t\$1"
> Output:
> ---
> | C1   | C2  | C3 | 
> ---
> | 1| 2017-03-21 13:31:18.0 | some data  | 
> ---
> #
> # STEP 02 - Import and Export without '$' (baseline)
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> "T1" --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
> hdfs dfs -cat /user/root/t1/part*
> Output:
> 17/03/21 06:39:31 INFO mapreduce.ImportJobBase: Transferred 34 bytes in 
> 21.6129 seconds (1.5731 bytes/sec)
> 17/03/21 06:39:31 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> 1,2017-03-21 13:30:02.0,some data
> sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> T1 --export-dir /user/root/t1 --num-mappers 1
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
> "select * from t1"
> Output:
> 17/03/21 06:46:07 INFO mapreduce.ExportJobBase: Transferred 163 bytes in 
> 21.4938 seconds (7.5836 bytes/sec)
> 17/03/21 06:46:07 INFO mapreduce.ExportJobBase: Exported 1 records.
> ~
> ---
> | C1   | C2  | C3 | 
> ---
> | 1| 2017-03-21 13:30:02.0 | some data  | 
> | 1| 2017-03-21 13:30:02.0 | some data  | 
> ---
> #
> # STEP 03 - Export with '$' (--table T$1)
> #
> sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> T$1 --export-dir /user/root/t1 --num-mappers 1
> Output:
> 17/03/21 06:49:28 INFO manager.SqlManager: Executing SQL statement: SELECT 
> t.* FROM "T" t WHERE 1=0
> 17/03/21 06:49:28 ERROR manager.SqlManager: Error executing statement: 
> java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
> #
> # STEP 04 - Export with '$' (--table T\$1)
> #
> sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table 
> T\$1 --export-

Re: Review Request 57876: Sqoop (export + --table) with Oracle table_name having '$' fails with error (ORA-00942 or java.lang.NoClassDefFoundError)

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57876/#review170140
---


Ship it!




Ship It!

- Anna Szonyi


On March 24, 2017, 1:20 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57876/
> ---
> 
> (Updated March 24, 2017, 1:20 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3159
> https://issues.apache.org/jira/browse/SQOOP-3159
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The fix itself could be done by rewriting line 
> org/apache/sqoop/orm/CompilationManager.java:303 from
> 
> int indexOfDollarSign = chompedPath.indexOf("$");
> 
> to
> 
> int indexOfDollarSign = chompedPath.lastIndexOf("$");
> 
> because that block is supposed to get the outer class name from the class 
> name but a class name could contain a $ sign as well.
> So for example the outer class name of class Foo$Bar$1 is be Foo$Bar.
> 
> However to make the unit testing easier and make the purpose of that method 
> clearer I did some refactoring in 
> org.apache.sqoop.orm.CompilationManager#addClassFilesFromDir method.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/CompilationManager.java c1a656b 
>   src/test/com/cloudera/sqoop/TestExport.java df5a663 
>   src/test/com/cloudera/sqoop/manager/OracleExportTest.java ec56cbe 
>   
> src/test/org/apache/sqoop/manager/oracle/OracleSpecialCharacterTableImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/orm/TestCompilationManager.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57876/diff/3/
> 
> 
> Testing
> ---
> 
> ant clean test
> 
> New unit tests and third party tests are added and ran successfully.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 57876: Sqoop (export + --table) with Oracle table_name having '$' fails with error (ORA-00942 or java.lang.NoClassDefFoundError)

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57876/#review169886
---


Fix it, then Ship it!




Hey Szabi, 

The changes look great, it's much more testable this way and easier to read and 
thanks for adding a new Import Test case. I have a few questions I posted below.

Thanks,
Anna


src/java/org/apache/sqoop/orm/CompilationManager.java
Lines 330-334 (patched)
<https://reviews.apache.org/r/57876/#comment242596>

Would it make sense to explicitly separate these into an ~isInnerClass vs 
~isOuterClass path of handling and make the flow more explicit based on this 
decisions?



src/java/org/apache/sqoop/orm/CompilationManager.java
Lines 338 (patched)
<https://reviews.apache.org/r/57876/#comment242586>

Would it make sense to extract this into a method, like isInnerClass to 
make it more explicit?


- Anna Szonyi


On March 23, 2017, 2:21 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57876/
> ---
> 
> (Updated March 23, 2017, 2:21 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3159
> https://issues.apache.org/jira/browse/SQOOP-3159
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The fix itself could be done by rewriting line 
> org/apache/sqoop/orm/CompilationManager.java:303 from
> 
> int indexOfDollarSign = chompedPath.indexOf("$");
> 
> to
> 
> int indexOfDollarSign = chompedPath.lastIndexOf("$");
> 
> because that block is supposed to get the outer class name from the class 
> name but a class name could contain a $ sign as well.
> So for example the outer class name of class Foo$Bar$1 is be Foo$Bar.
> 
> However to make the unit testing easier and make the purpose of that method 
> clearer I did some refactoring in 
> org.apache.sqoop.orm.CompilationManager#addClassFilesFromDir method.
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/CompilationManager.java c1a656b 
>   src/test/com/cloudera/sqoop/TestExport.java df5a663 
>   src/test/com/cloudera/sqoop/manager/OracleExportTest.java ec56cbe 
>   
> src/test/org/apache/sqoop/manager/oracle/OracleSpecialCharacterTableImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/orm/TestCompilationManager.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57876/diff/2/
> 
> 
> Testing
> ---
> 
> ant clean test
> 
> New unit tests and third party tests are added and ran successfully.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Sqoop CI changes

[Anna Szonyi]

Hi All,

I would like to make the following changes to the Sqoop CI system:
Disable the SCM polling for the Sqoop-hadoop23 Sqoop-hadoop20 and
Sqoop-hadoop100 jobs (and later delete the jobs themselves),
as the current trunk version of sqoop no longer contains these profiles, so
these runs are redundant.

I would also like to propose the creation of a pre-commit hook for Sqoop
(like the existing one for Sqoop2).

Please let me know if you have any objections.

Thanks,
Anna

[jira] [Commented] (SQOOP-3143) Restore fail messages removed in SQOOP-3091

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928804#comment-15928804
 ] 

Anna Szonyi commented on SQOOP-3143:


Thanks for this contribution Bogi!

/Anna

> Restore fail messages removed in SQOOP-3091
> ---
>
> Key: SQOOP-3143
> URL: https://issues.apache.org/jira/browse/SQOOP-3143
> Project: Sqoop
>  Issue Type: Sub-task
>Affects Versions: 1.4.6
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Minor
> Attachments: SQOOP-3143.patch
>
>
> Restore fail messages on excepted exceptions using 
> reportMissingExceptionWithMessage functionality added in JUnit 4.12



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57659: SQOOP-3143: Restore fail messages removed in SQOOP-3091

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57659/#review169185
---


Ship it!




Ship It!

- Anna Szonyi


On March 16, 2017, 6:23 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57659/
> ---
> 
> (Updated March 16, 2017, 6:23 p.m.)
> 
> 
> Review request for Sqoop, Attila Szabo and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3143
> https://issues.apache.org/jira/browse/SQOOP-3143
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Restored fail messages removed in SQOOP-3091 using 
> reportMissingExceptionWithMessage()
> 
> 
> Diffs
> -
> 
>   src/test/com/cloudera/sqoop/TestAvroExport.java 
> 487c12b96cb1ea9bd3f879997dc5746f347cd13f 
>   src/test/com/cloudera/sqoop/TestConnFactory.java 
> 62035cb54a46adc03d83a0923bddd68238b1f08e 
>   src/test/com/cloudera/sqoop/TestExportUpdate.java 
> 81f3a686cde521652d7f161f42de9dc94507c688 
>   src/test/com/cloudera/sqoop/TestParquetExport.java 
> 3c3db33b94971062b0f4ca2a0b4437b8df620d79 
>   src/test/com/cloudera/sqoop/TestSqoopOptions.java 
> 7d34d33951d09bdf4db8d48e94e5b0db55ef702d 
>   src/test/com/cloudera/sqoop/TestTargetDir.java 
> ded539231801d1e6853180b55a8f649a791e11e6 
>   src/test/org/apache/sqoop/TestExportUsingProcedure.java 
> b4b46f23da7668667c37ed8067e5ed304ef60b64 
> 
> 
> Diff: https://reviews.apache.org/r/57659/diff/1/
> 
> 
> Testing
> ---
> 
> ant clean test, ant test
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-3154) Sqoop2: Add Anna Szonyi to committer list in our pom file

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928527#comment-15928527
 ] 

Anna Szonyi commented on SQOOP-3154:


Thanks [~abrahamfine]!

Everything is *perfect*!

Thanks,
/Anna

> Sqoop2: Add Anna Szonyi to committer list in our pom file
> -
>
> Key: SQOOP-3154
> URL: https://issues.apache.org/jira/browse/SQOOP-3154
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Abraham Fine
>Assignee: Abraham Fine
> Fix For: 1.99.8
>
> Attachments: SQOOP-3154.patch
>
>
> Now that [~szonyi] is committer we should update our committer list in the 
> root pom.xml file:
> Also add [~abrahamfine] as PMC member.
> https://github.com/apache/sqoop/blob/sqoop2/pom.xml#L903



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 55769: Import tables with special characters

[Anna Szonyi]

> On March 8, 2017, 12:40 p.m., Attila Szabo wrote:
> > Hey Dimitry, Szabolcs, Anna,
> > 
> > First of all I'd like to thank Dimitry for opening this ticket, and 
> > providing a contribution idea how to solve this problem.
> > I'd like to also thank to Anna and Szabi to review the code of Dimitry.
> > 
> > Although I'd like to raise a two concerns around:
> > - I would do this one level above (on the SqoopOptions level) to ensure the 
> > mapping reflects the very same names that the engine will use for internal 
> > column name presentation.
> > - I would also introduce a new cmd line option for this mechanism (e.g. 
> > --escapeMappingColumnNames ), thus ensure we do not alter the default 
> > behaviour and thus don't have to worry about backward incompatibility and 
> > breaking changes. (maybe later we could alter the default behaviour by 
> > switching the default state of the boolean property)
> > 
> > Please change the implementation accordingly,
> > 
> > Thanks,
> > Attila
> > 
> > ps: Dimitry! I'd like to kindly ask you to add me back to the reviewers of 
> > this ticket, thus I would be able to spot it on my reviewboard dashboard, 
> > thus you don't have to wait such a long time, to get your ticket 
> > reviewed/committed. Many thanks in advance!

Hi Attila,

Thanks for adding your comments, it's a great catch for the multiple calls for 
cleanMapColumnJava, we hadn't noticed that!
However I would disagree with your second comment of introducing a new cmd line 
option, as I believe this is a backwards compatible change (also works with the 
old way of specifying of C_1 along with the correct C#1) - though we could put 
this debate behind with a simple unit/integration test :)

Would you say that if Dmitry fixes the cleanMapColumnJava and adds a test to 
prove backwards compatibility you would be comfortable committing this change?

Thanks,
/Anna


- Anna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55769/#review168288
---


On March 8, 2017, 8:27 p.m., Dmitry Zagorulkin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55769/
> ---
> 
> (Updated March 8, 2017, 8:27 p.m.)
> 
> 
> Review request for Sqoop, Attila Szabo, Olivier Lamy, and vishnu  s nair.
> 
> 
> Bugs: SQOOP-3123
> https://issues.apache.org/jira/browse/SQOOP-3123
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Special characters processing in table and column names
> 
> https://issues.apache.org/jira/browse/SQOOP-3123
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/ClassWriter.java c18a36f3 
>   src/test/com/cloudera/sqoop/TestAvroImport.java 26edd4ce 
> 
> 
> Diff: https://reviews.apache.org/r/55769/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dmitry Zagorulkin
> 
>

[jira] [Created] (SQOOP-3146) Sqoop (import + --as-parquetfile) with Oracle (CLOB vs. BLOB) is inconsistent

[Anna Szonyi (JIRA)]

Anna Szonyi created SQOOP-3146:
--

 Summary: Sqoop (import + --as-parquetfile) with Oracle (CLOB vs. 
BLOB) is inconsistent
 Key: SQOOP-3146
 URL: https://issues.apache.org/jira/browse/SQOOP-3146
 Project: Sqoop
  Issue Type: Bug
Reporter: Anna Szonyi



# Owner: Sqoopinators
# Component: Sqoop1
# Purpose: Escalation Test Case
# SFDC Case ID:127558
# SFDC EscalationID: CDH-50699
# File: SupportTest_Case_127558_JIRA_CDH-50699.txt
#
# Description
# 1. Sqoop import + —as-parquetfile + CLOB Data Types (Gives Error)
# 2. Sqoop import + —as-parquetfile + BLOB Data Types (Works Good)


##
 USE CASE [1] . Sqoop import + —as-parquetfile + CLOB Data Types (Gives Error)
##

###
# STEP 01 - CREATE DATA
###


sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
"create table t1_clob (c1 int,c2 clob)"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
"insert into t1_clob values(1,'qwqewewqrerew121212121212’)”
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query 
"select * from t1_clob"

#
OUTPUT
#

---
| C1   | C2   | 
---
| 1| qwqewewqrerew121212121212 | 
---



STEP 02 - IMPORT AS PARQUET FILE (Without —map-column-java) [REPRODUCING 
THE ERROR]



sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD 
--as-parquetfile --table T1_CLOB --delete-target-dir --target-dir 
'/projects/t1_clob' -m 1

OUTPUT
——

Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/02/21 10:07:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
17/02/21 10:07:08 WARN tool.BaseSqoopTool: Setting your password on the 
command-line is insecure. Consider using -P instead.
17/02/21 10:07:08 INFO oracle.OraOopManagerFactory: Data Connector for Oracle 
and Hadoop is disabled.
17/02/21 10:07:08 INFO manager.SqlManager: Using default fetchSize of 1000
17/02/21 10:07:08 INFO tool.CodeGenTool: Beginning code generation
17/02/21 10:07:08 INFO tool.CodeGenTool: Will generate java class as 
codegen_T1_CLOB
17/02/21 10:07:09 INFO manager.OracleManager: Time zone has been set to GMT
17/02/21 10:07:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* 
FROM "T1_CLOB" t WHERE 1=0
17/02/21 10:07:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is 
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: 
/tmp/sqoop-root/compile/cbaf5013e6bc9dad7283090f9d761289/codegen_T1_CLOB.java 
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/02/21 10:07:11 INFO orm.CompilationManager: Writing jar file: 
/tmp/sqoop-root/compile/cbaf5013e6bc9dad7283090f9d761289/codegen_T1_CLOB.jar
17/02/21 10:07:13 INFO tool.ImportTool: Destination directory /projects/t1_clob 
is not present, hence not deleting.
17/02/21 10:07:13 INFO manager.OracleManager: Time zone has been set to GMT
17/02/21 10:07:13 INFO manager.OracleManager: Time zone has been set to GMT
17/02/21 10:07:13 INFO mapreduce.ImportJobBase: Beginning import of T1_CLOB
17/02/21 10:07:13 INFO Configuration.deprecation: mapred.jar is deprecated. 
Instead, use mapreduce.job.jar
17/02/21 10:07:13 INFO manager.OracleManager: Time zone has been set to GMT
17/02/21 10:07:13 INFO manager.OracleManager: Time zone has been set to GMT
17/02/21 10:07:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* 
FROM "T1_CLOB" t WHERE 1=0
17/02/21 10:07:13 INFO manager.SqlManager: Executing SQL statement: SELECT t.* 
FROM "T1_CLOB" t WHERE 1=0
17/02/21 10:07:13 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL 
type 2005


#
STEP 02.1 - IMPORT AS PARQUET FILE + —map-column-java (For CLOB data type)
#

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD 
--as-parquet file --table T1_CLOB --map-column-java C2=String 
--delete-target-dir --target-dir '/projects/t1_clob' -m 1

OUTPUT
———
17/02/21 10:09:50 INFO tool.ImportTool: Destination directory /projects/t1_clob 
is not present, hence not deleting.
17/02/21 10:09:50 INFO mapreduce.ImportJobBase: Beginning import of T1_CLOB
17/02/21 10:10:09 INFO mapreduce.Job:  map 0% reduce 0%
17/02/21 10:10:34 INFO ma

[jira] [Assigned] (SQOOP-3145) Misc. test changes, so we don't need to fork with every test

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi reassigned SQOOP-3145:
--

Assignee: Anna Szonyi

> Misc. test changes, so we don't need to fork with every test
> 
>
> Key: SQOOP-3145
> URL: https://issues.apache.org/jira/browse/SQOOP-3145
> Project: Sqoop
>  Issue Type: Improvement
>        Reporter: Anna Szonyi
>    Assignee: Anna Szonyi
>
> Currently we can only run tests with "fork with every" single test. We can 
> make a few changes for the tests to be able to run without needing to fork 
> every tests (table names can contain the testnames, queryname.jar can be 
> overwritten, etc.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SQOOP-3145) Misc. test changes, so we don't need to fork with every test

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3145:
---
Description: Currently we can only run tests with "fork with every" single 
test. We can make a few changes for the tests to be able to run without needing 
to fork every tests (table names can contain the testnames, queryname.jar can 
be overwritten, etc.)

> Misc. test changes, so we don't need to fork with every test
> 
>
> Key: SQOOP-3145
> URL: https://issues.apache.org/jira/browse/SQOOP-3145
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Anna Szonyi
>
> Currently we can only run tests with "fork with every" single test. We can 
> make a few changes for the tests to be able to run without needing to fork 
> every tests (table names can contain the testnames, queryname.jar can be 
> overwritten, etc.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (SQOOP-3145) Misc. test changes, so we don't need to fork with every test

[Anna Szonyi (JIRA)]

Anna Szonyi created SQOOP-3145:
--

 Summary: Misc. test changes, so we don't need to fork with every 
test
 Key: SQOOP-3145
 URL: https://issues.apache.org/jira/browse/SQOOP-3145
 Project: Sqoop
  Issue Type: Improvement
Reporter: Anna Szonyi






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (SQOOP-3136) Sqoop should work well with not default file systems

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885723#comment-15885723
 ] 

Anna Szonyi commented on SQOOP-3136:


Hey [~yalovyyi],

Understandable, most of the time so do I, unfortunately Sqoop still 
supports/uses Java 7 - we are planning to propose a switch to java 8 in the 
near future (that will have to be a community decision), in the mean time we're 
stuck with 7. We have source and target enforcements in  ant, though it is to 
enforce the related syntax more than the compiler/compilation itself afair. 

[~maugli] added a small patch to remove the offending method, so there is not 
much else to do :).

Thanks again for the contribution!

Thanks,
Anna

> Sqoop should work well with not default file systems
> 
>
> Key: SQOOP-3136
> URL: https://issues.apache.org/jira/browse/SQOOP-3136
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors/hdfs
>Affects Versions: 1.4.5
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: SQOOP-3136.patch
>
>
> Currently Sqoop assumes default file system when it comes to IO operations. 
> It makes it hard to use other FileSystem implementations as source or 
> destination. Here is an example:
> {code}
> sqoop import --connect  --table table1 --driver  DRIVER> --username root --password  --delete-target-dir --target-dir 
> s3a://some-bucket/tmp/sqoop
> ...
> 17/02/15 19:16:59 ERROR tool.ImportTool: Imported Failed: Wrong FS: 
> s3a://some-bucket/tmp/sqoop, expected: hdfs://:8020
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (SQOOP-3136) Sqoop should work well with not default file systems

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882404#comment-15882404
 ] 

Anna Szonyi commented on SQOOP-3136:


Hi [~yalovyyi] [~maugli],

Thanks for the patch, this is super useful! However I am/HUDSON is having 
issues running the tests - 
/src/test/org/apache/sqoop/util/TestFileSystemUtil.java:36: error: cannot find 
symbol
conf.set("fs.my.impl", MyFileSystem.class.getTypeName());
As far as I know [this 
method|https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getTypeName--]
 was introduced in Java 1.8 and currently Sqoop supports/is built with/for 1.7 
which is causing the above issue.

Could you please look into this?

Thanks,
Anna

> Sqoop should work well with not default file systems
> 
>
> Key: SQOOP-3136
> URL: https://issues.apache.org/jira/browse/SQOOP-3136
> Project: Sqoop
>  Issue Type: Improvement
>  Components: connectors/hdfs
>Affects Versions: 1.4.5
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: SQOOP-3136.patch
>
>
> Currently Sqoop assumes default file system when it comes to IO operations. 
> It makes it hard to use other FileSystem implementations as source or 
> destination. Here is an example:
> {code}
> sqoop import --connect  --table table1 --driver  DRIVER> --username root --password  --delete-target-dir --target-dir 
> s3a://some-bucket/tmp/sqoop
> ...
> 17/02/15 19:16:59 ERROR tool.ImportTool: Imported Failed: Wrong FS: 
> s3a://some-bucket/tmp/sqoop, expected: hdfs://:8020
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 56984: SQOOP-3139: change default of mapreduce.map.maxattempts to 1 for import (so it's consistent with export)

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56984/
---

Review request for Sqoop, Boglarka Egyed, Attila Szabo, Szabolcs Vasas, and Liz 
Szilagyi.


Bugs: SQOOP-3139
https://issues.apache.org/jira/browse/SQOOP-3139


Repository: sqoop-trunk


Description
---

SQOOP-3139: sqoop tries to re-execute select query during import in case of a 
connection reset error and this is causing lots of duplicate records from source


Diffs
-

  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 105917c 

Diff: https://reviews.apache.org/r/56984/diff/


Testing
---

ant test


Thanks,

Anna Szonyi

[jira] [Updated] (SQOOP-3139) sqoop tries to re execute select query during import in case of a connection reset error and this is causing lots of duplicate records from source

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3139:
---
Attachment: SQOOP-3139.patch

> sqoop tries to re execute select query during import in case of a connection 
> reset error and this is causing lots of duplicate records from source
> --
>
> Key: SQOOP-3139
> URL: https://issues.apache.org/jira/browse/SQOOP-3139
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: IBM Hadoop distribution: 4.2.0
> version: 1.4.6_IBM_27
> Sqoop 1.4.6_IBM_27
>Reporter: hemanth meka
> Attachments: SQOOP-3139.patch
>
>
> We cannot reproduce this issue as it depends on network. Here is a edited log 
> content for understanding the issue. 
> Log start
> .
> .
> 2017-02-22 07:35:37,638 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:35:37,662 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:39:00,533 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>   at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>   at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>   at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>   at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>   at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>   at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>   at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>   at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>   at QueryResult.readFields0(QueryResult.java:10706)
>   at QueryResult.readFields(QueryResult.java:10415)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>   at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> 2017-02-22 07:39:00,552 WARN [main] 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader: Trying to recover from 
> DB read

[jira] [Commented] (SQOOP-3139) sqoop tries to re execute select query during import in case of a connection reset error and this is causing lots of duplicate records from source

[Anna Szonyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880362#comment-15880362
 ] 

Anna Szonyi commented on SQOOP-3139:


Hi,

This might be because of the mapper task re-tries the failed action 4 times by 
default. There was a fix for this (SQOOP-2055) for the export case. Something 
similar should probably also be done for the import. 

I've created a patch on top of SQOOP-3140 (note the patch also needs 3140 to be 
committed to be useable) for the import to set the mapred max attempt to one. 
I would like to propose this change to the community: that we set the same 
default of 1 attempt for export and import, with the possibility of overriding 
it as needed.

In the mean time you can try overriding the mapreduce.map.maxattempts to 1 
instead of the default 4.

Best,
/Anna

> sqoop tries to re execute select query during import in case of a connection 
> reset error and this is causing lots of duplicate records from source
> --
>
> Key: SQOOP-3139
> URL: https://issues.apache.org/jira/browse/SQOOP-3139
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: IBM Hadoop distribution: 4.2.0
> version: 1.4.6_IBM_27
> Sqoop 1.4.6_IBM_27
>Reporter: hemanth meka
>
> We cannot reproduce this issue as it depends on network. Here is a edited log 
> content for understanding the issue. 
> Log start
> .
> .
> 2017-02-22 07:35:37,638 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:35:37,662 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:39:00,533 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>   at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>   at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>   at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>   at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>   at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>   at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>   at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>   at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>   at QueryResult.readFields0(QueryResult.java:10706)
>   at QueryResult.readFields(QueryResult.java:10415)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>   at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.

[jira] [Assigned] (SQOOP-3139) sqoop tries to re execute select query during import in case of a connection reset error and this is causing lots of duplicate records from source

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi reassigned SQOOP-3139:
--

Assignee: Anna Szonyi

> sqoop tries to re execute select query during import in case of a connection 
> reset error and this is causing lots of duplicate records from source
> --
>
> Key: SQOOP-3139
> URL: https://issues.apache.org/jira/browse/SQOOP-3139
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: IBM Hadoop distribution: 4.2.0
> version: 1.4.6_IBM_27
> Sqoop 1.4.6_IBM_27
>Reporter: hemanth meka
>Assignee: Anna Szonyi
> Attachments: SQOOP-3139.patch
>
>
> We cannot reproduce this issue as it depends on network. Here is a edited log 
> content for understanding the issue. 
> Log start
> .
> .
> 2017-02-22 07:35:37,638 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:35:37,662 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 1000 )
> 2017-02-22 07:39:00,533 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>   at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>   at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>   at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>   at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>   at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>   at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>   at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>   at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>   at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>   at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>   at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>   at QueryResult.readFields0(QueryResult.java:10706)
>   at QueryResult.readFields(QueryResult.java:10415)
>   at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>   at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>   at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>   at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>   at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> 2017-02-22 07:39:00,552 WARN [main] 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader: 

Re: Review Request 56949: SQOOP-3140: mapred.map.max.attempts, mapred.reduce.max.attempts are deprecated

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56949/
---

(Updated Feb. 23, 2017, 10:52 a.m.)


Review request for Sqoop, Boglarka Egyed, Attila Szabo, Szabolcs Vasas, and Liz 
Szilagyi.


Changes
---

extraceted to constants as it should have been :)


Bugs: SQOOP-3140
https://issues.apache.org/jira/browse/SQOOP-3140


Repository: sqoop-trunk


Description
---

SQOOP-3140: mapred.map.max.attempts, mapred.reduce.max.attempts are deprecated. 
Instead, use mapreduce.map.maxattempts, mapreduce.reduce.maxattempts


Diffs (updated)
-

  src/java/org/apache/sqoop/manager/oracle/OraOopConstants.java 26f1901 
  src/java/org/apache/sqoop/manager/oracle/OraOopManagerFactory.java 17873bc 
  src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 27f84da 
  src/java/org/apache/sqoop/mapreduce/JobBase.java 256d4f7 
  src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 
c0c6039 

Diff: https://reviews.apache.org/r/56949/diff/


Testing
---

ant test


Thanks,

Anna Szonyi

Re: Review Request 55769: Import tables with special characters

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55769/#review166495
---



Hi Dmitry,

Thanks for the patch, looks good! I would second Szabolcs's comment that it 
would be good to add a test case that covers the "backwards compatibility": 
using the cleaned column name for mapping with non-avro cases - we checked it 
out and it shouldn't be a problem, but it would be great to have automated test 
cases for it.

Thanks,
/Anna

- Anna Szonyi


On Jan. 27, 2017, 1:54 p.m., Dmitry Zagorulkin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55769/
> ---
> 
> (Updated Jan. 27, 2017, 1:54 p.m.)
> 
> 
> Review request for Sqoop, Olivier Lamy and vishnu  s nair.
> 
> 
> Bugs: SQOOP-3123
> https://issues.apache.org/jira/browse/SQOOP-3123
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Special characters processing in table and column names
> 
> https://issues.apache.org/jira/browse/SQOOP-3123
> 
> 
> Diffs
> -
> 
>   src/java/org/apache/sqoop/orm/ClassWriter.java c18a36f3 
>   src/test/com/cloudera/sqoop/TestAvroImport.java 26edd4ce 
> 
> Diff: https://reviews.apache.org/r/55769/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dmitry Zagorulkin
> 
>

Review Request 56949: SQOOP-3140: mapred.map.max.attempts, mapred.reduce.max.attempts are deprecated

[Anna Szonyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56949/
---

Review request for Sqoop, Boglarka Egyed, Attila Szabo, Szabolcs Vasas, and Liz 
Szilagyi.


Bugs: SQOOP-3140
https://issues.apache.org/jira/browse/SQOOP-3140


Repository: sqoop-trunk


Description
---

SQOOP-3140: mapred.map.max.attempts, mapred.reduce.max.attempts are deprecated. 
Instead, use mapreduce.map.maxattempts, mapreduce.reduce.maxattempts


Diffs
-

  src/java/org/apache/sqoop/manager/oracle/OraOopConstants.java 26f1901 
  src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 27f84da 
  src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 
c0c6039 

Diff: https://reviews.apache.org/r/56949/diff/


Testing
---

ant test


Thanks,

Anna Szonyi

[jira] [Updated] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3140:
---
Attachment: SQOOP-3140.patch

> mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 
> - old property is used by SQOOP-2055
> --
>
> Key: SQOOP-3140
> URL: https://issues.apache.org/jira/browse/SQOOP-3140
> Project: Sqoop
>  Issue Type: Bug
>    Reporter: Anna Szonyi
>    Assignee: Anna Szonyi
> Attachments: SQOOP-3140.patch
>
>
> mapred.map.max.attempts is 
> deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
>  We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3140:
---
Attachment: (was: SQOOP-3104.patch)

> mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 
> - old property is used by SQOOP-2055
> --
>
> Key: SQOOP-3140
> URL: https://issues.apache.org/jira/browse/SQOOP-3140
> Project: Sqoop
>  Issue Type: Bug
>    Reporter: Anna Szonyi
>    Assignee: Anna Szonyi
>
> mapred.map.max.attempts is 
> deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
>  We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3140:
---
Attachment: SQOOP-3104.patch

> mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 
> - old property is used by SQOOP-2055
> --
>
> Key: SQOOP-3140
> URL: https://issues.apache.org/jira/browse/SQOOP-3140
> Project: Sqoop
>  Issue Type: Bug
>    Reporter: Anna Szonyi
>    Assignee: Anna Szonyi
> Attachments: SQOOP-3104.patch
>
>
> mapred.map.max.attempts is 
> deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
>  We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi reassigned SQOOP-3140:
--

Assignee: Anna Szonyi

> mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 
> - old property is used by SQOOP-2055
> --
>
> Key: SQOOP-3140
> URL: https://issues.apache.org/jira/browse/SQOOP-3140
> Project: Sqoop
>  Issue Type: Bug
>    Reporter: Anna Szonyi
>    Assignee: Anna Szonyi
>
> mapred.map.max.attempts is 
> deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
>  We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Szonyi updated SQOOP-3140:
---
Description: mapred.map.max.attempts is 
deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
 We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.

> mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 
> - old property is used by SQOOP-2055
> --
>
> Key: SQOOP-3140
> URL: https://issues.apache.org/jira/browse/SQOOP-3140
> Project: Sqoop
>  Issue Type: Bug
>    Reporter: Anna Szonyi
>
> mapred.map.max.attempts is 
> deprecated(http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html).
>  We need to instead, use mapreduce.map.maxattempts for the SQOOP-2055 fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (SQOOP-3140) mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts - old property is used by SQOOP-2055

[Anna Szonyi (JIRA)]

Anna Szonyi created SQOOP-3140:
--

 Summary: mapred.map.max.attempts is deprecated. Instead, use 
mapreduce.map.maxattempts - old property is used by SQOOP-2055
 Key: SQOOP-3140
 URL: https://issues.apache.org/jira/browse/SQOOP-3140
 Project: Sqoop
  Issue Type: Bug
Reporter: Anna Szonyi






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Import more than 10 million records from MySQL to HDFS

[Anna Szonyi]

Hi Wenxing,

I tested the scenario with a simple table (10 million rows, but only 3
columns) and it seems to work perfectly for me (with super simple types:
int, varchar, timestamp).
Could you please share the full create table statement (doesn't have to be
the same column names just the same types) and some sample inserts to see
what causes the problem, as it doesn't seem to be the row size. It maybe to
do with the types or the number of columns.
Also please share your link and job setup & the exception you got.

Thanks,
/Anna

On Mon, Jan 16, 2017 at 6:59 PM, wenxing zheng 
wrote:

> Hi Szabolcs,
>
> Sorry for the late reply. From my test, it's ok for 100, rows.
>
> Thanks, Wenxing
>
> On Wed, Jan 11, 2017 at 12:42 AM, Szabolcs Vasas 
> wrote:
>
> > Hi Wenxing,
> >
> > I have created a table based on the column information you sent but I
> won't
> > be able to do this testing in the next couple of days.
> > Btw have you tried the import with smaller data sets? I mean have you
> tried
> > to test what is the biggest data set you can import successfully?
> >
> > Szabolcs
> >
> > On Wed, Jan 4, 2017 at 10:55 AM, wenxing zheng 
> > wrote:
> >
> > > Hi Szabolcs,
> > >
> > > I am testing this scenario with our client's slave database. And I am
> > > sorry that I can not share the table definition and the sample data
> here.
> > > But attached is a sample of table definition with the column types.
> > >
> > > It's quite complex.
> > >
> > > Thanks, Wenxing
> > >
> > > On Wed, Jan 4, 2017 at 4:24 PM, Szabolcs Vasas 
> > wrote:
> > >
> > >> Hi Wenxing,
> > >>
> > >> I haven't tried this scenario yet but I would be happy to test it on
> my
> > >> side. Can you please send me the DDL statement for creating the MySQL
> > >> table
> > >> and some sample data?
> > >> Also it would be very helpful to send the details of the job you would
> > >> like
> > >> to run.
> > >>
> > >> Regards,
> > >> Szabolcs
> > >>
> > >> On Wed, Jan 4, 2017 at 2:54 AM, wenxing zheng <
> wenxing.zh...@gmail.com>
> > >> wrote:
> > >>
> > >> > can anyone help to advice?
> > >> >
> > >> > And I met with a problem when I set the checkColumn with
> updated_time,
> > >> but
> > >> > currently all the updated_time are in NULL. Under this case, the
> Sqoop
> > >> will
> > >> > fail to start the job. I think we need to support such kind of case.
> > >> >
> > >> > On Thu, Dec 29, 2016 at 9:18 AM, wenxing zheng <
> > wenxing.zh...@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Dear all,
> > >> > >
> > >> > > Did anyone already try to import more than 10 million data from
> > MySQL
> > >> to
> > >> > > HDFS by using the Sqoop2?
> > >> > >
> > >> > > I always failed at the very beginning with various throttling
> > >> settings,
> > >> > > but never made it.
> > >> > >
> > >> > > Appreciated for any advice.
> > >> > > Thanks, Wenxing
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Szabolcs Vasas
> > >> Software Engineer
> > >> 
> > >>
> > >
> > >
> >
> >
> > --
> > Szabolcs Vasas
> > Software Engineer
> > 
> >
>

Re: Review Request 55142: SQOOP-3099 Remove all junit3 TestCase extension and test suites

[Anna Szonyi]

/sqoop/mapreduce/db/TestIntegerSplitter.java e93b6ad 
  src/test/org/apache/sqoop/mapreduce/db/TestTextSplitter.java c402a54 
  
src/test/org/apache/sqoop/mapreduce/db/TextSplitterHadoopConfIntegrationTest.java
 32ebf45 
  src/test/org/apache/sqoop/tool/TestBaseSqoopTool.java fbbffe9 
  src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 3e502d0 
  src/test/org/apache/sqoop/validation/AbortOnFailureHandlerTest.java f38164c 
  src/test/org/apache/sqoop/validation/AbsoluteValidationThresholdTest.java 
9ac5074 
  src/test/org/apache/sqoop/validation/RowCountValidatorImportTest.java 035d3b1 

Diff: https://reviews.apache.org/r/55142/diff/


Testing
---

ant clean test, ant clean test -Dthirdparty=true 
-Dsqoop.thirdparty.lib.dir=3rdpartylib 
-Dsqoop.test.mysql.connectstring.host_url=mysqlserver 
-Dsqoop.test.mysql.databasename=sqoop -Dsqoop.test.mysql.password=sqoop 
-Dsqoop.test.mysql.username=sqoop 
-Dsqoop.test.oracle.connectstring=oracleserver 
-Dsqoop.test.postgresql.connectstring.host_url=postgresqlserver 
-Dsqoop.test.cubrid.connectstring.host_url=cubridserver 
-Dsqoop.test.cubrid.connectstring.username=sqoop 
-Dsqoop.test.cubrid.connectstring.database=sqoop 
-Dsqoop.test.cubrid.connectstring.password=sqoop 
-Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" 
-Dtest.timeout=100


Thanks,

Anna Szonyi