Re: [DISCUSS] Deprecate fuse-dfs
Hi folks, Recently we started investigating fuse-dfs for use of ML frameworks on k8s. I think that it is convenient way for data scientist to support access to HDFS via FUSE. We also expect that we do not need to implement a function for HDFS access each ML frameworks. I guess it is common use-case of HDFS for ML users. If there is no maintainer in the community, my staffs is capable to take on the role. By the way, I wonder, - How do you access to data in HDFS from k8s? Any other solution rather than fuse-dfs? - How do you think about another implementation for fuse like https://github.com/Microsoft/hdfs-mount . Should we ignore fuse-dfs and use or implement the other? Regards, Shingo On 2018/10/01 20:10:07, Wei-Chiu Chuang wrote: > Hi fellow Hadoop developers,> > > I want to start this thread to raise the awareness of the quality of> > fuse-dfs. It appears that this sub-component is not being developed and> > maintained, and appears not many are using it.> > > In the past two years, there has been only one bug fixed (HDFS-13322).> > > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS)%20AND%20text%20~%20fuse%20ORDER%20BY%20created%20DESC%2C%20updated%20DESC> > > > It doesn't support keytab login, ACL permissions, rename, ... a number of> > POSIX semantics. We also recently realized fuse-dfs doesn't work under> > heavy weight workload (Think running SQL applications on it)> > > So what's the status now? Is there any one who is still using fuse-dfs in> > production? Should we start the deprecation process? Or at least document> > that it is not meant for anything beyond simple data transfer? IIRC vim> > would even complain if you try to edit a file in fuse_dfs directory.> > -- > > A very happy Hadoop contributor> > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop Ozone 0.2.1-alpha release
It gives me great pleasure to announce that the Apache Hadoop community has voted to release Apache Hadoop Ozone 0.2.1-alpha. Apache Hadoop Ozone is an object store for Hadoop built using Hadoop Distributed Data Store. For more information and to download, please check https://hadoop.apache.org/ozone Note: This release is alpha quality, it's not recommended to use in production. Many thanks to everyone who contributed to the release, and everyone in the Apache Hadoop community! The release is a result of work from many contributors. Thank you for all of them. On behalf of the Hadoop community, Márton Elek ps: Hadoop Ozone and HDDS are released separately from the main Hadoop releases, this release doesn't include new Hadoop Yarn/Mapreduce/Hdfs versions. - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15809) ABFS: better exception handling when making getAccessToken call
Da Zhou created HADOOP-15809: Summary: ABFS: better exception handling when making getAccessToken call Key: HADOOP-15809 URL: https://issues.apache.org/jira/browse/HADOOP-15809 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Currently getAccessToken throws only IOException and it is never checked for cases like 401, 403, which lead to unnecessary retry, this should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [IMPORTANT] Apache JIRA doesn't properly show sub-tickets from parent
Just checked the JIRA again, now the issue has gone. I think it might be caused by some intermittent JIRA system issue. Thanks, Wangda On Mon, Oct 1, 2018 at 10:52 AM Wangda Tan wrote: > Hi all devs, > > Today I found many subtickets doesn't show properly under parent. > > For example, YARN-6875 is the parent of YARN-7072. But from YARN-6875's > sub ticket, YARN-7072 isn't shown here. > > I found many sub-tickets are gone from parent JIRA, such as YARN-6223 > (there were ~20+ sub tickets), YARN-2492 (There were ~70-80 sub tickets). > > Are there any changes recently to Apache JIRA? > > Thanks, > Wangda >
[DISCUSS] Deprecate fuse-dfs
Hi fellow Hadoop developers, I want to start this thread to raise the awareness of the quality of fuse-dfs. It appears that this sub-component is not being developed and maintained, and appears not many are using it. In the past two years, there has been only one bug fixed (HDFS-13322). https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS)%20AND%20text%20~%20fuse%20ORDER%20BY%20created%20DESC%2C%20updated%20DESC It doesn't support keytab login, ACL permissions, rename, ... a number of POSIX semantics. We also recently realized fuse-dfs doesn't work under heavy weight workload (Think running SQL applications on it) So what's the status now? Is there any one who is still using fuse-dfs in production? Should we start the deprecation process? Or at least document that it is not meant for anything beyond simple data transfer? IIRC vim would even complain if you try to edit a file in fuse_dfs directory. -- A very happy Hadoop contributor
[jira] [Created] (HADOOP-15808) Harden Token service loader use
Steve Loughran created HADOOP-15808: --- Summary: Harden Token service loader use Key: HADOOP-15808 URL: https://issues.apache.org/jira/browse/HADOOP-15808 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.9.1, 3.1.2 Reporter: Steve Loughran Assignee: Steve Loughran The Hadoop token service loading (identifiers, renewers...) works provided there's no problems loading any registered implementation. If there's a classloading or classcasting problem, the exception raised will stop all token support working; possibly the application not starting. This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on the classpath. It probably lurks in the wasb/abfs support too, but things have worked there because the installations with DT support there have always had correctly set up classpaths. Fix: do what we did for the FS service loader. Catch failures to instantiate a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[IMPORTANT] Apache JIRA doesn't properly show sub-tickets from parent
Hi all devs, Today I found many subtickets doesn't show properly under parent. For example, YARN-6875 is the parent of YARN-7072. But from YARN-6875's sub ticket, YARN-7072 isn't shown here. I found many sub-tickets are gone from parent JIRA, such as YARN-6223 (there were ~20+ sub tickets), YARN-2492 (There were ~70-80 sub tickets). Are there any changes recently to Apache JIRA? Thanks, Wangda
[jira] [Created] (HADOOP-15807) ITestS3AContractRootDir failure on non-S3Guarded bucket
Steve Loughran created HADOOP-15807: --- Summary: ITestS3AContractRootDir failure on non-S3Guarded bucket Key: HADOOP-15807 URL: https://issues.apache.org/jira/browse/HADOOP-15807 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.3.0 Reporter: Steve Loughran Got a root test failure against S3 London, possibly consistency related. The abstract test case should use eventually() here -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15806) Move ITestS3AMiniYarnCluster to S3A committers
Steve Loughran created HADOOP-15806:
---
Summary: Move ITestS3AMiniYarnCluster to S3A committers
Key: HADOOP-15806
URL: https://issues.apache.org/jira/browse/HADOOP-15806
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.2.0
Reporter: Steve Loughran
Only just realised from a stack trace, {{ITestS3AMiniYarnCluster}} is using
file output, and not the new S3A Committers. Fix: move
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 123] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.containermanager.TestNMProxy hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-compile-javac-root.txt [300K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-patch-pylint.txt [40K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/913/artifact/out/diff-javadoc-javadoc-root.txt [756K] CTEST:
