[jira] [Created] (HADOOP-16147) Allow CopyListing sequence file keys and values to be more easily customized
Andrew Olson created HADOOP-16147:
-
Summary: Allow CopyListing sequence file keys and values to be
more easily customized
Key: HADOOP-16147
URL: https://issues.apache.org/jira/browse/HADOOP-16147
Project: Hadoop Common
Issue Type: Improvement
Components: tools/distcp
Reporter: Andrew Olson
We have encountered a scenario where, when using the Crunch library to run a
distributed copy (CRUNCH-660, CRUNCH-675) at the conclusion of a job we need to
dynamically rename target paths to the preferred destination output part file
names, rather than retaining the original source path names.
A custom CopyListing implementation appears to be the proper solution for this.
However the place where the current SimpleCopyListing logic needs to be
adjusted is in a private method (writeToFileListing), so a relatively large
portion of the class would need to be cloned.
To minimize the amount of code duplication required for such a custom
implementation, we propose adding two new protected methods to the CopyListing
class, that can be used to change the actual keys and/or values written to the
copy listing sequence file:
{noformat}
protected Text getFileListingKey(Path sourcePathRoot, CopyListingFileStatus
fileStatus);
protected CopyListingFileStatus getFileListingValue(CopyListingFileStatus
fileStatus);
{noformat}
The SimpleCopyListing class would then be modified to consume these methods as
follows,
{noformat}
fileListWriter.append(
getFileListingKey(sourcePathRoot, fileStatus),
getFileListingValue(fileStatus));
{noformat}
The default implementations would simply preserve the present behavior of the
SimpleCopyListing class, and could reside in either CopyListing or
SimpleCopyListing, whichever is preferable.
{noformat}
protected Text getFileListingKey(Path sourcePathRoot, CopyListingFileStatus
fileStatus) {
return new Text(DistCpUtils.getRelativePath(sourcePathRoot,
fileStatus.getPath()));
}
protected CopyListingFileStatus getFileListingValue(CopyListingFileStatus
fileStatus) {
return fileStatus;
}
{noformat}
Please let me know if this proposal seems to be on the right track. If so I can
provide a patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/ [Feb 24, 2019 4:31:39 AM] (aengineer) HDDS-1166. Fix checkstyle line length issues. Contributed by Nandakumar. [Feb 24, 2019 4:45:01 AM] (aengineer) HDDS-1167. Error in hadoop-ozone/dev-support/checks/checkstyle.sh. [Feb 24, 2019 4:56:24 AM] (aengineer) HDDS-1147. Fix failing unit tests in TestOzoneManager. Contributed by [Feb 24, 2019 5:13:47 AM] (aengineer) HDDS-1155.Fix failing unit test methods of TestDeadNodeHandler. [Feb 24, 2019 5:37:55 AM] (github) HDDS-1120. Add a config to disable checksum verification during read. [Feb 24, 2019 9:07:06 AM] (sunilg) YARN-8821. [YARN-8851] GPU hierarchy/topology scheduling support based [Feb 24, 2019 2:17:29 PM] (wwei) YARN-9300. Lazy preemption should trigger an update on queue preemption [Feb 24, 2019 2:42:27 PM] (wwei) YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID [Feb 24, 2019 8:36:03 PM] (aengineer) HDDS-1168.Use random ports in TestBlockManager and TestDeletedBlockLog. [Feb 24, 2019 9:13:37 PM] (aengineer) HDDS-1154. Add tracing to the client side of [Feb 24, 2019 9:22:36 PM] (aengineer) HDDS-1149. Change the default ozone.client.checksum.type. Contributed by [Feb 24, 2019 9:53:38 PM] (aengineer) HDDS-1089. Disable OzoneFSStorageStatistics for hadoop versions older [Feb 24, 2019 10:02:30 PM] (aengineer) HDDS-1090. Add a recipe to the ozone docs about using Spark, OzoneFS, [Feb 24, 2019 10:40:52 PM] (aengineer) HDDS-1115. Provide ozone specific top-level pom.xml. Contributed by [Feb 24, 2019 11:04:25 PM] (aengineer) HDDS-1145. Add optional web server to the Ozone freon test tool. [Feb 24, 2019 11:20:34 PM] (aengineer) HDDS-1152. Add trace information for the client side of the datanode [Feb 24, 2019 11:39:41 PM] (aengineer) HDDS-1070. Adding Node and Pipeline related metrics in SCM. Contributed -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.util.TestDiskCheckerWithDiskIo hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash hadoop.hdfs.server.namenode.TestNamenodeCapacityReport hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem hadoop.mapred.TestJobCounters hadoop.yarn.sls.TestSLSGenericSynth cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-compile-javac-root.txt [336K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-patch-hadolint.txt [8.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-patch-pylint.txt [144K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/whitespace-eol.txt [9.6M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1058/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trun
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-common-project/hadoop-common Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient non-serializable instance field map In GlobalStorageStatistics.java:instance field map In GlobalStorageStatistics.java FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Dead store to state in org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream, INodeSymlink) At FSImageFormatPBINode.java:org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream, INodeSymlink) At FSImageFormatPBINode.java:[line 623] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.TestEncryptionZonesWithKMS hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMProxy cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-compile-cc-root-jdk1.8.0_191.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-compile-javac-root-jdk1.8.0_191.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/xml.txt [8.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/243/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc:
[jira] [Created] (HADOOP-16146) Make start-build-env.sh safe in case of misusage of DOCKER_INTERACTIVE_RUN
Elek, Marton created HADOOP-16146:
-
Summary: Make start-build-env.sh safe in case of misusage of
DOCKER_INTERACTIVE_RUN
Key: HADOOP-16146
URL: https://issues.apache.org/jira/browse/HADOOP-16146
Project: Hadoop Common
Issue Type: Bug
Reporter: Elek, Marton
Assignee: Elek, Marton
[~aw] reported the problem in HDDS-891:
{quote}DOCKER_INTERACTIVE_RUN opens the door for users to set command line
options to docker. Most notably, -c and -v and a few others that share one
particular characteristic: they reference the file system. As soon as shell
code hits the file system, it is no longer safe to assume space delimited
options. In other words, -c /My Cool Filesystem/Docker Files/config.json or -v
/c_drive/Program Files/Data:/data may be something a user wants to do, but the
script now breaks because of the IFS assumptions.
{quote}
DOCKER_INTERACTIVE_RUN was used in jenkins to run normal build process in
docker. In case of DOCKER_INTERACTIVE_RUN was set to empty the docker container
is started without the "-i -t" flags.
It can be improved by checking the value of the environment variable and enable
only fixed set of values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
