[jira] [Created] (HADOOP-13983) Print better error when accessing a non-existent store
John Zhuge created HADOOP-13983: --- Summary: Print better error when accessing a non-existent store Key: HADOOP-13983 URL: https://issues.apache.org/jira/browse/HADOOP-13983 Project: Hadoop Common Issue Type: Bug Components: fs/adl Affects Versions: 3.0.0-alpha2 Reporter: John Zhuge Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13982) Print better error message when accessing a store without permission
John Zhuge created HADOOP-13982:
---
Summary: Print better error message when accessing a store without
permission
Key: HADOOP-13982
URL: https://issues.apache.org/jira/browse/HADOOP-13982
Project: Hadoop Common
Issue Type: Bug
Components: fs/adl
Affects Versions: 3.0.0-alpha2
Reporter: John Zhuge
The error message when accessing a store without permission is not user
friendly:
{noformat}
$ hdfs dfs -ls adl://STORE.azuredatalakestore.net/
ls: Operation GETFILESTATUS failed with HTTP403 : null
{noformat}
Store {{STORE}} exists but Hadoop is configured with an SPI that does not have
access to the store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13981) S3Guard CLI: Add documentation
Aaron Fabbri created HADOOP-13981: - Summary: S3Guard CLI: Add documentation Key: HADOOP-13981 URL: https://issues.apache.org/jira/browse/HADOOP-13981 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri I believe we still need documentation for the new S3Guard CLI commands. Synopsis of all the commands and some examples would be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13980) S3Guard CLI: Add fsck check command
Aaron Fabbri created HADOOP-13980: - Summary: S3Guard CLI: Add fsck check command Key: HADOOP-13980 URL: https://issues.apache.org/jira/browse/HADOOP-13980 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri As discussed in HADOOP-13650, we want to add an S3Guard CLI command which compares S3 with MetadataStore, and returns a failure status if any invariants are violated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13979) Live unit tests leaks files in home dir on ADLS store
John Zhuge created HADOOP-13979:
---
Summary: Live unit tests leaks files in home dir on ADLS store
Key: HADOOP-13979
URL: https://issues.apache.org/jira/browse/HADOOP-13979
Project: Hadoop Common
Issue Type: Bug
Components: fs/adl
Affects Versions: 3.0.0-alpha2
Reporter: John Zhuge
Priority: Minor
Live unit tests left 61 files in user home dir on ADLS store {{jzadls}}:
{noformat}
/user
/user/jzhuge
/user/jzhuge/06b74549-c9d5-41b3-9f32-660e3284200d
/user/jzhuge/0b71b60d-7501-40b2-a86c-c1ed2542997f
/user/jzhuge/1311d721-8a31-4eda-9d5b-be4fc47ce62a
...
{noformat}
However, failed to reproduce on store {{jzhugeadls}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13978) Update project release notes for 3.0.0-alpha2
Andrew Wang created HADOOP-13978: Summary: Update project release notes for 3.0.0-alpha2 Key: HADOOP-13978 URL: https://issues.apache.org/jira/browse/HADOOP-13978 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha2 Reporter: Andrew Wang Assignee: Andrew Wang Let's update the website release notes for 3.0.0-alpha2's changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13977) IntelliJ Compilation error in ITUseMiniCluster.java
Miklos Szegedi created HADOOP-13977: --- Summary: IntelliJ Compilation error in ITUseMiniCluster.java Key: HADOOP-13977 URL: https://issues.apache.org/jira/browse/HADOOP-13977 Project: Hadoop Common Issue Type: Bug Reporter: Miklos Szegedi Assignee: Sean Busbey The repro steps: mvn clean install -DskipTests and then "Build/Build Project" in IntelliJ IDEA to update indexes, etc. ...hadoop/hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java Error:(34, 28) java: package org.apache.hadoop.fs does not exist ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13976) Path globbing does not match newlines
Eric Badger created HADOOP-13976: Summary: Path globbing does not match newlines Key: HADOOP-13976 URL: https://issues.apache.org/jira/browse/HADOOP-13976 Project: Hadoop Common Issue Type: Bug Reporter: Eric Badger Assignee: Eric Badger Need to add the DOTALL flag to allow for newlines to be accepted as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13975) Allow DistCp to use MultiThreadedMapper
Zheng Shao created HADOOP-13975: --- Summary: Allow DistCp to use MultiThreadedMapper Key: HADOOP-13975 URL: https://issues.apache.org/jira/browse/HADOOP-13975 Project: Hadoop Common Issue Type: New Feature Components: tools/distcp Affects Versions: 3.0.0-alpha1 Reporter: Zheng Shao Assignee: Zheng Shao Priority: Minor Although distcp allow users to control the parallelism via number of mappers, sometimes it's desirable to run fewer mappers but more threads per mapper. Since distcp is network bound (either by throughput or more frequently by latency of creating connections, opening files, reading/writing files, and closing files), this can make each mapper much more efficient. In that way, a lot of resources can be shared so we can save memory and connections to NameNode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/ [Jan 10, 2017 6:43:11 PM] (arp) HDFS-11299. Support multiple Datanode File IO hooks. Contributed by [Jan 10, 2017 7:32:48 PM] (wang) HDFS-11072. Add ability to unset and change directory EC policy. [Jan 11, 2017 2:20:27 AM] (junping_du) YARN-6079. Fix simple spelling errors in yarn test code. Contributed by [Jan 11, 2017 4:12:42 AM] (mingma) HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj [Jan 11, 2017 8:26:02 AM] (Arun Suresh) HADOOP-13903. Improvements to KMS logging to help debug authorization -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestBlockStoragePolicy hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapreduce.TestMRJobClient hadoop.tools.util.TestProducerConsumer Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-compile-root.txt [120K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-compile-root.txt [120K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-compile-root.txt [120K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/214/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K]
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/ [Jan 10, 2017 8:38:01 AM] (sunilg) YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs [Jan 10, 2017 10:05:01 AM] (naganarasimha_gr) YARN-5937. stop-yarn.sh is not able to gracefully stop node managers. [Jan 10, 2017 10:24:16 AM] (naganarasimha_gr) YARN-6054. TimelineServer fails to start when some LevelDb state files [Jan 10, 2017 11:37:58 AM] (lei) HDFS-11259. Update fsck to display maintenance state info. (Manoj [Jan 10, 2017 6:43:11 PM] (arp) HDFS-11299. Support multiple Datanode File IO hooks. Contributed by [Jan 10, 2017 7:32:48 PM] (wang) HDFS-11072. Add ability to unset and change directory EC policy. [Jan 11, 2017 2:20:27 AM] (junping_du) YARN-6079. Fix simple spelling errors in yarn test code. Contributed by [Jan 11, 2017 4:12:42 AM] (mingma) HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestReadStripedFileWithMissingBlocks hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 hadoop.hdfs.TestMaintenanceState hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestRMAdminService hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.applications.distributedshell.TestDistributedShell cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [420K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/283/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits
Steve Loughran created HADOOP-13974: --- Summary: S3a CLI to support list/purge of pending multipart commits Key: HADOOP-13974 URL: https://issues.apache.org/jira/browse/HADOOP-13974 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Steve Loughran The S3A CLI will need to be able to list and delete pending multipart commits. We can do the cleanup already via fs.s3a properties. The CLI will let scripts stat for outstanding data (have a different exit code) and permit batch jobs to explicitly trigger cleanups. This will become critical with the multipart committer, as there's a significantly higher likelihood of commits remaining outstanding. We may also want to be able to enumerate/cancel all pending commits in the FS tree -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13912) S3a Multipart Committer (avoid rename)
[
https://issues.apache.org/jira/browse/HADOOP-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13912.
-
Resolution: Duplicate
closing as duplicate of HADOOP-1786; adding subjiras there
> S3a Multipart Committer (avoid rename)
> --
>
> Key: HADOOP-13912
> URL: https://issues.apache.org/jira/browse/HADOOP-13912
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Object stores do not have an efficient rename operation, which is used by the
> Hadoop FileOutputCommitter to atomically promote the "winning" attempt out of
> the multiple (speculative) attempts to the final path. These slow job commits
> are one of the main friction points when using object stores in Hadoop.There
> have been quite some attempts at resolving this: HADOOP-9565, Apache Spark
> DirectOutputCommitters, ... but they have proven not to be robust in face of
> adversity (network partitions, ...).
> The current ticket proposes to do the atomic commit by using the S3 Multipart
> API, which allows multiple concurrent uploads on the same objectname, each in
> its own "temporary space, identified by the UploadId which is returned as a
> response to InitiateMultipartUpload. Every attempt writes directly to the
> final {{outputPath}}. Data is uploaded using Put Part and as a response an
> ETag for the part is returned and stored. The CompleteMultipartUpload is
> postponed. Instead, we persist the UploadId (using a _temporary subdir or
> elsewhere) and the ETags. When a certain "job" wins
> {{CompleteMultipartUpload}} is called for each of its files using the proper
> list of Part ETags.
> Completing a MultipartUpload is a metadata only operation (internally in S3)
> and is thus orders of magnitude faster than the rename-based approach which
> moves all the data.
> Required work:
> * Expose the multipart initiate and complete calls in S3AOutputStream to
> S3AFilesystem
> * Use these multipart calls in a custom committer as described above. I
> propose to build on the S3ACommitter [~ste...@apache.org] is doing for
> HADOOP-13786
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13973) S3 requests failing: java.lang.IllegalStateException: Connection is not open
Steve Loughran created HADOOP-13973: --- Summary: S3 requests failing: java.lang.IllegalStateException: Connection is not open Key: HADOOP-13973 URL: https://issues.apache.org/jira/browse/HADOOP-13973 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.8.0 Environment: EC2 cluster Reporter: Rajesh Balamohan Assignee: Steve Loughran S3 requests failing with an error coming from Http client, "java.lang.IllegalStateException: Connection is not open" Some online discussion implies that this is related to shared connection pool shutdown & fixed in http client 4.4+. Hadoop & AWS SDK use v 4.5.2 so the fix is in, we just need to make sure the pool is being set up right. There's a problem here of course: it may require moving to a later version of the AWS SDK, with the consequences on jackson , as seen in HADOOP-13050. And that's if there is a patched version out there -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
