[jira] [Commented] (SQOOP-3224) Mainframe FTP transfer should have an option to use binary mode for transfer
[ https://issues.apache.org/jira/browse/SQOOP-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515282#comment-16515282 ] ASF GitHub Bot commented on SQOOP-3224: --- Github user christeoh closed the pull request at: https://github.com/apache/sqoop/pull/47 > Mainframe FTP transfer should have an option to use binary mode for transfer > > > Key: SQOOP-3224 > URL: https://issues.apache.org/jira/browse/SQOOP-3224 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > > Currently the mainframe FTP module is hard coded to use ascii transfer mode. > Propose a mainframe module flag to be able to change modes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sqoop pull request #47: SQOOP-3224: Mainframe Binary File Transfer mode
Github user christeoh closed the pull request at: https://github.com/apache/sqoop/pull/47 ---
Re: Review Request 62492: SQOOP-3224: Mainframe FTP transfer should have an option to use binary mode for transfer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62492/ --- (Updated June 18, 2018, 11:47 a.m.) Review request for Sqoop. Changes --- Added integration tests. Bugs: SQOOP-3224 https://issues.apache.org/jira/browse/SQOOP-3224 Repository: sqoop-trunk Description --- Added --as-binaryfile and --buffersize to support FTP transfer mode switching. Diffs (updated) - build.xml 0ae729bc src/docs/user/import-mainframe.txt abeb7cde src/java/org/apache/sqoop/SqoopOptions.java d9984af3 src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 3b542102 src/java/org/apache/sqoop/mapreduce/KeyRecordWriters.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java fec34f21 src/java/org/apache/sqoop/mapreduce/mainframe/AbstractMainframeDatasetImportMapper.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java ea54b07f src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryImportMapper.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryRecord.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java 1f78384b src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java 0b7b5b85 src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 8ef30d38 src/java/org/apache/sqoop/tool/BaseSqoopTool.java c62ee98c src/java/org/apache/sqoop/tool/ImportTool.java 2c474b7e src/java/org/apache/sqoop/tool/MainframeImportTool.java 8883301d src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java 95bc0ecb src/test/org/apache/sqoop/manager/mainframe/MainframeManagerImportTest.java 041dfb78 src/test/org/apache/sqoop/manager/mainframe/MainframeTestUtil.java f28ff36c src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java PRE-CREATION src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 3547294f src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 0b0c6c34 src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java 90a85194 Diff: https://reviews.apache.org/r/62492/diff/17/ Changes: https://reviews.apache.org/r/62492/diff/16-17/ Testing --- Unit tests. Functional testing on mainframe. Thanks, Chris Teoh
[jira] [Commented] (SQOOP-3319) Extract code using Kite into separate classes
[ https://issues.apache.org/jira/browse/SQOOP-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515051#comment-16515051 ] Szabolcs Vasas commented on SQOOP-3319: --- Linking SQOOP-3331 as it fixes an issue introduced by this JIRA. > Extract code using Kite into separate classes > - > > Key: SQOOP-3319 > URL: https://issues.apache.org/jira/browse/SQOOP-3319 > Project: Sqoop > Issue Type: Sub-task >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 3.0.0 > > Attachments: SQOOP-3319.patch, SQOOP-3319.patch > > > Kite Dataset API is used in many places in the code to read/write Parquet > files and configure MR jobs. > The goal of this JIRA is to introduce an implementation agnostic interface > for Parquet reading/writing and extract the code using Kite Dataset API into > separate classes implementing this interface. The benefit of this refactoring > is that it enables us introducing a new Parquet reading/writing > implementation which does not use Kite but plugs in easily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3313) Remove Kite dependency
[ https://issues.apache.org/jira/browse/SQOOP-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515049#comment-16515049 ] Szabolcs Vasas commented on SQOOP-3313: --- Linking SQOOP-3331 as it adds a test and fixes a Mainframe connector problem added by SQOOP-3319. > Remove Kite dependency > -- > > Key: SQOOP-3313 > URL: https://issues.apache.org/jira/browse/SQOOP-3313 > Project: Sqoop > Issue Type: Improvement >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Fix For: 3.0.0 > > > Having Kite as a dependency makes it hard to release a version of Sqoop > compatible with Hadoop 3. > For details see discussion on dev list in [this thread|http://example.com] > and also SQOOP-3305. > Let's use this ticket to gather features that need to be > changed/reimplemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3331) Add Mainframe FTP integration test for GDG dataset.
[ https://issues.apache.org/jira/browse/SQOOP-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3331. --- Resolution: Fixed Fix Version/s: 3.0.0 Hi [~chris.t...@gmail.com], Your patch is committed, thank you for your contribution! Regards, Szabolcs > Add Mainframe FTP integration test for GDG dataset. > --- > > Key: SQOOP-3331 > URL: https://issues.apache.org/jira/browse/SQOOP-3331 > Project: Sqoop > Issue Type: Test >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Fix For: 3.0.0 > > Attachments: SQOOP-3331.patch > > > Current mainframe import functionality doesn't have integration tests, only > unit tests. Adding this test improves validation of this functionality. > Dockerfile and accompanying contents at > https://github.com/christeoh/zos-ftpmock/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3331) Add Mainframe FTP integration test for GDG dataset.
[ https://issues.apache.org/jira/browse/SQOOP-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3331: -- Attachment: SQOOP-3331.patch > Add Mainframe FTP integration test for GDG dataset. > --- > > Key: SQOOP-3331 > URL: https://issues.apache.org/jira/browse/SQOOP-3331 > Project: Sqoop > Issue Type: Test >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Attachments: SQOOP-3331.patch > > > Current mainframe import functionality doesn't have integration tests, only > unit tests. Adding this test improves validation of this functionality. > Dockerfile and accompanying contents at > https://github.com/christeoh/zos-ftpmock/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
