[jira] [Created] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus
Ted Yu created HADOOP-15831:
---
Summary: Include modificationTime in the toString method of
CopyListingFileStatus
Key: HADOOP-15831
URL: https://issues.apache.org/jira/browse/HADOOP-15831
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
I was looking at a DistCp error observed in hbase backup test:
{code}
2018-10-08 18:12:03,067 WARN [Thread-933] mapred.LocalJobRunner$Job(590):
job_local1175594345_0004
java.io.IOException: Inconsistent sequence file: current chunk file
org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
length = 5142 aclEntries = null, xAttrs = null}
at
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
at
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
2018-10-08 18:12:03,150 INFO [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask:
1.0 mapProgress: 1.0
{code}
I noticed that modificationTime was not included in the toString of
CopyListingFileStatus.
I propose including modificationTime so that it is easier to tell when the
respective files last change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15828) Review of MachineList class
BELUGA BEHR created HADOOP-15828:
Summary: Review of MachineList class
Key: HADOOP-15828
URL: https://issues.apache.org/jira/browse/HADOOP-15828
Project: Hadoop Common
Issue Type: Improvement
Components: util
Affects Versions: 3.2.0
Reporter: BELUGA BEHR
Attachments: HADOOP-15828.1.patch
Clean up and simplify class {{MachineList}}. Primarily, remove LinkedList
implementation and use empty collections instead of 'null' values, add logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15827) NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth s3guard
Steve Loughran created HADOOP-15827:
---
Summary: NPE in DynamoDBMetadataStore.lambda$listChildren for root
+ auth s3guard
Key: HADOOP-15827
URL: https://issues.apache.org/jira/browse/HADOOP-15827
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.2.0
Reporter: Steve Loughran
NPE in a test run of {{-Dparallel-tests -DtestsThreadCount=6 -Ds3guard
-Ddynamodb -Dauthoritative}}
{code}
[ERROR] testLSRootDir(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
Time elapsed: 42.822 s <<< ERROR!
java.lang.NullPointerException
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15193) add bulk delete call to metastore API & DDB impl
[ https://issues.apache.org/jira/browse/HADOOP-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15193. - Resolution: Won't Fix Fix Version/s: 3.3.0 > add bulk delete call to metastore API & DDB impl > > > Key: HADOOP-15193 > URL: https://issues.apache.org/jira/browse/HADOOP-15193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > > recursive dir delete (and any future bulk delete API like HADOOP-15191) > benefits from using the DDB bulk table delete call, which takes a list of > deletes and executes. Hopefully this will offer better perf. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15826) @Retries annotation of putObject() call & uses wrong
Steve Loughran created HADOOP-15826: --- Summary: @Retries annotation of putObject() call & uses wrong Key: HADOOP-15826 URL: https://issues.apache.org/jira/browse/HADOOP-15826 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.2.0 Reporter: Steve Loughran Assignee: Steve Loughran The retry annotations of the S3AFilesystem putObject call and its writeOperationsHelper use aren't in sync with what the code does. Fix -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/ [Oct 7, 2018 7:48:30 AM] (aajisaka) YARN-8788. mvn package -Pyarn-ui fails on JDK9 -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy hadoop.hdfs.server.namenode.TestFSImage hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.web.TestWebHDFS hadoop.hdfs.server.namenode.TestBackupNode hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-compile-javac-root.txt [300K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-patch-pylint.txt [40K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/whitespace-eol.txt [9.3M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/diff-javadoc-javadoc-root.txt [756K] CTEST: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/920/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt [116K] unit:
