[
https://issues.apache.org/jira/browse/SQOOP-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364676#comment-14364676
]
Hudson commented on SQOOP-2201:
-------------------------------
SUCCESS: Integrated in Sqoop2-hadoop200 #785 (See
[https://builds.apache.org/job/Sqoop2-hadoop200/785/])
SQOOP-2201: Sqoop2: Add possibility to read Hadoop configuration files to HFDS
connector (abe:
https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=6ca31c5054331d74ba9b099c87a33bf8c081afb1)
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/IncrementalReadTest.java
*
test/src/test/java/org/apache/sqoop/integration/server/SubmissionWithDisabledModelObjectsTest.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/FromRDBMSToHDFSTest.java
* connector/connector-hdfs/src/main/resources/hdfs-connector-config.properties
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
* test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/PartitionerTest.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsUtils.java
*
test/src/test/java/org/apache/sqoop/integration/connector/kafka/FromHDFSToKafkaTest.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/TestHdfsConnector.java
* test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/FromHDFSToRDBMSTest.java
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/AllTypesTest.java
*
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/TableStagedRDBMSTest.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
*
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/LinkConfig.java
> Sqoop2: Add possibility to read Hadoop configuration files to HFDS connector
> ----------------------------------------------------------------------------
>
> Key: SQOOP-2201
> URL: https://issues.apache.org/jira/browse/SQOOP-2201
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.99.5
> Reporter: Jarek Jarcec Cecho
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.6
>
> Attachments: SQOOP-2201.patch, SQOOP-2201.patch, SQOOP-2201.patch
>
>
> Currently the HDFS connector is not explicitly reading Hadoop configuration
> files. During
> [Initialization|https://github.com/apache/sqoop/blob/sqoop2/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java]
> phase it doesn't do anything, so the configuration files are not needed.
> During other parts of the workflow, we're [explicitly
> casting|https://github.com/apache/sqoop/blob/sqoop2/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java#L61]
> the general {{Context}} object to Hadoop {{Configuration}}.
> This is unfortunate because:
> * It couples HDFS connector to Mapreduce execution engine. It will break with
> adding non mapreduce based execution engine.
> * We can't do any HDFS specific checks in {{Initializer}} as the Hadoop
> {{Configuration}} object is not available there.
> As a result I would like to propose breaking this coupling between HDFS
> connector and Mapreduce execution engine and add configuration option to HDFS
> Link to specify directory from which we should read the appropriate Hadoop
> configuration files (with reasonable defaults such as {{/etc/conf/hadoop}}).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
