[jira] [Created] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
Shashikant Banerjee created HDFS-12340: -- Summary: Ozone: C/C++ implementation of ozone client using curl Key: HDFS-12340 URL: https://issues.apache.org/jira/browse/HDFS-12340 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 This Jira is introduced for implementation of ozone client in C/C++ using curl library. All these calls will make use of HTTP protocol and would require libcurl. The libcurl API are referenced from here: https://curl.haxx.se/libcurl/ Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Branch merges and 3.0.0-beta1 scope
For timeline service v2, we have completed all subtasks under YARN-5355 [1]. We initiated a merge-to-trunk vote [2] yesterday. thanks Vrushali [1] https://issues.apache.org/jira/browse/YARN-5355 [2] http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201708.mbox/%3CCAE=b_fblt2j+ezb4wqdn_uwbig1sd5kpqgaw+9br__zou5y...@mail.gmail.com%3E On Wed, Aug 23, 2017 at 11:12 AM, Vinod Kumar Vavilapalli < vino...@apache.org> wrote: > Agreed. I was very clearly not advocating for rushing in features. If you > have followed my past emails, I have only strongly advocated features be > worked in branches and get merged when they are in a reasonable state. > > Each branch contributor group should look at their readiness and merge > stuff in assuming that the branch reached a satisfactory state. That’s it. > > From release management perspective, blocking features just because we are > a month close to the deadline is not reasonable. Let the branch > contributors rationalize, make this decision and the rest of us can support > them in making the decision. > > +Vinod > > > At this point, there have been three planned alphas from September 2016 > until July 2017 to "get in features". While a couple of upcoming features > are "a few weeks" away, I think all of us are aware how predictable > software development schedules can be. I think we can also all agree that > rushing just to meet a release deadline isn't the best practice when it > comes to software development either. > > > > Andrew has been very clear about his goals at each step and I think > Wangda's willingness to not rush in resource types was an appropriate > response. I'm sympathetic to the goals of getting in a feature for 3.0, > but it might be a good idea for each project that is a "few weeks away" to > seriously look at the readiness compared to the features which have been > testing for 6+ months already. > > > > -Ray > >
Re: Branch merges and 3.0.0-beta1 scope
Agreed. I was very clearly not advocating for rushing in features. If you have followed my past emails, I have only strongly advocated features be worked in branches and get merged when they are in a reasonable state. Each branch contributor group should look at their readiness and merge stuff in assuming that the branch reached a satisfactory state. That’s it. From release management perspective, blocking features just because we are a month close to the deadline is not reasonable. Let the branch contributors rationalize, make this decision and the rest of us can support them in making the decision. +Vinod > At this point, there have been three planned alphas from September 2016 until > July 2017 to "get in features". While a couple of upcoming features are "a > few weeks" away, I think all of us are aware how predictable software > development schedules can be. I think we can also all agree that rushing > just to meet a release deadline isn't the best practice when it comes to > software development either. > > Andrew has been very clear about his goals at each step and I think Wangda's > willingness to not rush in resource types was an appropriate response. I'm > sympathetic to the goals of getting in a feature for 3.0, but it might be a > good idea for each project that is a "few weeks away" to seriously look at > the readiness compared to the features which have been testing for 6+ months > already. > > -Ray
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/ [Aug 22, 2017 8:14:12 AM] (aajisaka) YARN-7047. Moving logging APIs over to slf4j in [Aug 22, 2017 10:55:48 AM] (stevel) HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator. [Aug 22, 2017 2:47:39 PM] (xiao) HADOOP-14705. Add batched interface reencryptEncryptedKeys to KMS. [Aug 22, 2017 5:56:09 PM] (jlowe) YARN-2416. InvalidStateTransitonException in ResourceManager if [Aug 22, 2017 6:16:24 PM] (jlowe) YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. [Aug 22, 2017 9:50:01 PM] (jlowe) HADOOP-14687. AuthenticatedURL will reuse bad/expired session cookies. [Aug 23, 2017 2:20:57 AM] (subru) YARN-7053. Move curator transaction support to ZKCuratorManager. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:[line 490] Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.net.TestDNS hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.yarn.sls.nodemanager.TestNMSimulator Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/whitespace-tabs.txt [1.2M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/diff-javadoc-javadoc-root.txt [1.9M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [148K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [236K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/501/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [16K] Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12342) Differentiate webhdfs vs. swebhdfs calls in audit log
Erik Krogen created HDFS-12342:
--
Summary: Differentiate webhdfs vs. swebhdfs calls in audit log
Key: HDFS-12342
URL: https://issues.apache.org/jira/browse/HDFS-12342
Project: Hadoop HDFS
Issue Type: Improvement
Components: logging
Reporter: Erik Krogen
Assignee: Erik Krogen
Currently the audit log only logs {{webhdfs}} vs {{rpc}} as the {{proto}}. It
is useful to be able to audit whether certain commands were carried out via
webhdfs or swebhdfs as this has different security and potentially performance
implications. We have been running this internally for a while and have found
it useful for looking at usage patterns.
Proposal is just to continue logging {{webhdfs}} as the proto for {{http}}
WebHDFS commands, but log {{swebhdfs}} for SWebHDFS (over {{https}}). This will
be incompatible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Branch merges and 3.0.0-beta1 scope
Also, when people +1 a merge, can they please describe if they did testing / use the feature in addition to what is already described in the thread? On Wed, Aug 23, 2017 at 11:18 AM, Vrushali Channapattan < vrushalic2...@gmail.com> wrote: > For timeline service v2, we have completed all subtasks under YARN-5355 > [1]. > > We initiated a merge-to-trunk vote [2] yesterday. > > thanks > Vrushali > [1] https://issues.apache.org/jira/browse/YARN-5355 > [2] > http://mail-archives.apache.org/mod_mbox/hadoop-common- > dev/201708.mbox/%3CCAE=b_fbLT2J+Ezb4wqdN_UwBiG1Sd5kpqGaw+9Br__zou5yNTQ@ > mail.gmail.com%3E > > > On Wed, Aug 23, 2017 at 11:12 AM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > > > Agreed. I was very clearly not advocating for rushing in features. If you > > have followed my past emails, I have only strongly advocated features be > > worked in branches and get merged when they are in a reasonable state. > > > > Each branch contributor group should look at their readiness and merge > > stuff in assuming that the branch reached a satisfactory state. That’s > it. > > > > From release management perspective, blocking features just because we > are > > a month close to the deadline is not reasonable. Let the branch > > contributors rationalize, make this decision and the rest of us can > support > > them in making the decision. > > > > +Vinod > > > > > At this point, there have been three planned alphas from September 2016 > > until July 2017 to "get in features". While a couple of upcoming > features > > are "a few weeks" away, I think all of us are aware how predictable > > software development schedules can be. I think we can also all agree > that > > rushing just to meet a release deadline isn't the best practice when it > > comes to software development either. > > > > > > Andrew has been very clear about his goals at each step and I think > > Wangda's willingness to not rush in resource types was an appropriate > > response. I'm sympathetic to the goals of getting in a feature for 3.0, > > but it might be a good idea for each project that is a "few weeks away" > to > > seriously look at the readiness compared to the features which have been > > testing for 6+ months already. > > > > > > -Ray > > > > >
[jira] [Created] (HDFS-12343) TestKeys#testPutAndGetKeyWithDnRestart failed
Xiaoyu Yao created HDFS-12343: - Summary: TestKeys#testPutAndGetKeyWithDnRestart failed Key: HDFS-12343 URL: https://issues.apache.org/jira/browse/HDFS-12343 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7240 Reporter: Xiaoyu Yao Assignee: Mukul Kumar Singh The ticket is opened to fix the test that failed with a different error after HDFS-12216. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12344) LocatedFileStatus regression: no longer accepting null FSPermission
Ewan Higgs created HDFS-12344:
-
Summary: LocatedFileStatus regression: no longer accepting null
FSPermission
Key: HDFS-12344
URL: https://issues.apache.org/jira/browse/HDFS-12344
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ewan Higgs
Priority: Minor
SPARK-21817 was opened to fix a NPE in Spark where it calls
{{LocatedFileStatus}} with a null {{FSPermission}}. This breaks in current
HEAD. However, the {{LocatedFileStatus}} is a stable/evolving API so this is
actually a regression introduced in HDFS-6984.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
+1 (Binding), Great work guys, unwavering dedication and team effort. Kudos to the whole team. Though i have not tested, just went through the jiras and features added. Just wanted to know whether there was any other key out standing features left after phase 2 and their impacts ? And it would interesting to share the plans (if available) of upstream projects like Tez, Spark to incorporate ATSv2. Regards, + Naga On Thu, Aug 24, 2017 at 12:28 AM, varunsax...@apache.org < varun.saxena.apa...@gmail.com> wrote: > Hi All, > > Just to update. > Folks who may be interested in going through the documentation for the > feature can refer to current documentation attached in pdf format, on the > umbrella JIRA i.e. YARN-5355. > > Furthermore, we have run the YARN-5355 patch(with all the changes made in > the branch) against trunk and the build is almost green barring a few > checkstyle issues. The test failures which come up in the build are > outstanding issues on trunk. > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo > mmentId=16138266=com.atlassian.jira.plugin.system. > issuetabpanels:comment-tabpanel#comment-16138266 > > Thanks, > Varun Saxena. > > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan < > vrushalic2...@gmail.com> wrote: > > > Hi folks, > > > > Per earlier discussion [1], I'd like to start a formal vote to merge > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote > will > > run for 7 days, and will end August 29 11:00 PM PDT. > > > > We have previously completed one merge onto trunk [3] and Timeline > Service > > v2 has been part of Hadoop release 3.0.0-alpha1. > > > > Since then, we have been working on extending the capabilities of > Timeline > > Service v2 in a feature branch [2] for a while, and we are reasonably > > confident that the state of the feature meets the criteria to be merged > > onto trunk and we'd love folks to get their hands on it in a test > capacity > > and provide valuable feedback so that we can make it production-ready. > > > > In a nutshell, Timeline Service v.2 delivers significant scalability and > > usability improvements based on a new architecture. What we would like to > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a > > complete end-to-end read/write flow with security and read level > > authorization via whitelists. You should be able to start setting it up > and > > testing it. > > > > At a high level, the following are the key features that have been > > implemented since alpha1: > > - Security via Kerberos Authentication and delegation tokens > > - Read side simple authorization via whitelist > > - Client configurable entity sort ordering > > - Richer REST APIs for apps, app attempts, containers, fetching metrics > by > > timerange, pagination, sub-app entities > > - Support for storing sub-application entities (entities that exist > outside > > the scope of an application) > > - Configurable TTLs (time-to-live) for tables, configurable table > prefixes, > > configurable hbase cluster > > - Flow level aggregations done as dynamic (table level) coprocessors > > - Uses latest stable HBase release 1.2.6 > > > > There are a total of 82 subtasks that were completed as part of this > > effort. > > > > We paid close attention to ensure that once disabled Timeline Service v.2 > > does not impact existing functionality when disabled (by default). > > > > Special thanks to a team of folks who worked hard and contributed towards > > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > > > Regards, > > Vrushali > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html > > [2] https://issues.apache.org/jira/browse/YARN-5355 > > [3] https://issues.apache.org/jira/browse/YARN-2928 > > [4] https://github.com/apache/hadoop/commits/YARN-5355 > > >
[jira] [Created] (HDFS-12347) TestBalancerRPCDelay#testBalancerRPCDelay fails consistently
Xiao Chen created HDFS-12347:
Summary: TestBalancerRPCDelay#testBalancerRPCDelay fails
consistently
Key: HDFS-12347
URL: https://issues.apache.org/jira/browse/HDFS-12347
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.0.0-beta1
Reporter: Xiao Chen
Seems to be failing consistently on trunk from yesterday-ish.
A sample failure is
https://builds.apache.org/job/PreCommit-HDFS-Build/20824/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancerRPCDelay/testBalancerRPCDelay/
Running locally failed with:
{noformat}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Hi Naga Thank you for the kind words and appreciate your vote very much. Yes, among everyone who has worked on contributing to the TSv2 code, there has been a strong sense of camaraderie. Let me try to address your questions one by one. bq. Though i have not tested, just went through the jiras and features added. Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks, Huawei and Cloudera. bq. whether there was any other key out standing features left after phase 2 and their impacts ? Yes, we have some features that we would like to build into TSv2 and we have included those as part of "Current Status and Future Plans" in the documentation. Some of the future plans include: - Clustering of the readers - Migration and compatibility with v.1 - Timeline collectors as separate instances from node managers - Support for off-cluster clients - More robust storage fault tolerance - Better support for long-running apps - Support for ACLs - Offline (time-based periodic) aggregation for flows, users, and queues for reporting and analysis bq. it would interesting to share the plans (if available) of upstream projects like Tez, Spark to incorporate ATSv2. We've had an initial discussion with some of the Tez community members a few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were part of the discussion. It was a productive session and we came away with a set of things to do right away, of which we have already added in features like support for sub-application entities and simple whitelisting of users so that Tez community can start experimenting with it. I believe some of the Tez Yahoo! folks are in the very early stages of exploring TSv2. Rohith Sharma from Hortonworks has done some more groundwork and has some advanced thoughts around the TSv2-Tez integration considerations. We also have had an initial discussion with the Federation team (Subru, Carlo) a while back. Accordingly, we have already updated some aspects of our schema to accommodate things like the application to cluster mappings. We have an open jira for Federation integration (YARN-5357) which is not included in this version. We have not yet reached out to Spark folks but would love to do so. thanks Vrushali On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla < naganarasimha...@apache.org> wrote: > +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos > to the whole team. > Though i have not tested, just went through the jiras and features added. > > Just wanted to know whether there was any other key out standing features > left after phase 2 and their impacts ? > And it would interesting to share the plans (if available) of upstream > projects like Tez, Spark to incorporate ATSv2. > > Regards, > + Naga > > > > On Thu, Aug 24, 2017 at 12:28 AM, varunsax...@apache.org < > varun.saxena.apa...@gmail.com> wrote: > > > Hi All, > > > > Just to update. > > Folks who may be interested in going through the documentation for the > > feature can refer to current documentation attached in pdf format, on the > > umbrella JIRA i.e. YARN-5355. > > > > Furthermore, we have run the YARN-5355 patch(with all the changes made in > > the branch) against trunk and the build is almost green barring a few > > checkstyle issues. The test failures which come up in the build are > > outstanding issues on trunk. > > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo > > mmentId=16138266=com.atlassian.jira.plugin.system. > > issuetabpanels:comment-tabpanel#comment-16138266 > > > > Thanks, > > Varun Saxena. > > > > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan < > > vrushalic2...@gmail.com> wrote: > > > > > Hi folks, > > > > > > Per earlier discussion [1], I'd like to start a formal vote to merge > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote > > will > > > run for 7 days, and will end August 29 11:00 PM PDT. > > > > > > We have previously completed one merge onto trunk [3] and Timeline > > Service > > > v2 has been part of Hadoop release 3.0.0-alpha1. > > > > > > Since then, we have been working on extending the capabilities of > > Timeline > > > Service v2 in a feature branch [2] for a while, and we are reasonably > > > confident that the state of the feature meets the criteria to be merged > > > onto trunk and we'd love folks to get their hands on it in a test > > capacity > > > and provide valuable feedback so that we can make it production-ready. > > > > > > In a nutshell, Timeline Service v.2 delivers significant scalability > and > > > usability improvements based on a new architecture. What we would like > to > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a > > > complete end-to-end read/write flow with security and read level > > > authorization via whitelists. You should be able to start setting it up > > and > > > testing it. > > > > > > At a high level, the following are the key features that have been > > >
[jira] [Created] (HDFS-12348) disable removing block to trash while rolling upgrade
Jiandan Yang created HDFS-12348: Summary: disable removing block to trash while rolling upgrade Key: HDFS-12348 URL: https://issues.apache.org/jira/browse/HDFS-12348 Project: Hadoop HDFS Issue Type: New Feature Components: datanode Reporter: Jiandan Yang Assignee: Jiandan Yang DataNode remove block file and meta file to trash while rolling upgrade，and do delete when executing finalize. But frequently creating and deleting files leads to disk to be full（eg，Hbase compaction), and we will not rollbacking namespace in production when rolling upgrade fail. Disable trash of datanode maybe a good method to avoid disk to be full. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12343) TestKeys#testPutAndGetKeyWithDnRestart failed
[ https://issues.apache.org/jira/browse/HDFS-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDFS-12343. -- Resolution: Fixed Fix Version/s: HDFS-7240 This issue has been fixed with HDFS-12337. Duplicating it. > TestKeys#testPutAndGetKeyWithDnRestart failed > - > > Key: HDFS-12343 > URL: https://issues.apache.org/jira/browse/HDFS-12343 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > > The ticket is opened to fix the test that failed with a different error after > HDFS-12216. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12345) Scale testing HDFS NameNode
Zhe Zhang created HDFS-12345: Summary: Scale testing HDFS NameNode Key: HDFS-12345 URL: https://issues.apache.org/jira/browse/HDFS-12345 Project: Hadoop HDFS Issue Type: Bug Reporter: Zhe Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12113) `hadoop fs -setrep` requries huge amount of memory on client side
[
https://issues.apache.org/jira/browse/HDFS-12113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved HDFS-12113.
---
Resolution: Duplicate
[~Tagar] Thanks for reporting the issue. It is a dup of HADOOP-12502.
> `hadoop fs -setrep` requries huge amount of memory on client side
> -
>
> Key: HDFS-12113
> URL: https://issues.apache.org/jira/browse/HDFS-12113
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.6.5
> Environment: Java 7
>Reporter: Ruslan Dautkhanov
>
> {code}
> $ hadoop fs -setrep -w 3 /
> {code}
> was failing with
> {noformat}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2367)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> at java.lang.StringBuilder.append(StringBuilder.java:132)
> at
> org.apache.hadoop.fs.shell.PathData.getStringForChildPath(PathData.java:305)
> at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:272)
> at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
> at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
> at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
> at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
> at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at
> org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:76)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {noformat}
> Until hadoop fs cli command's Java heap memory was allowed to grow to 5Gb:
> {code}
> HADOOP_HEAPSIZE=5000 hadoop fs -setrep -w 3 /
> {code}
> Notice that this setrep change was done for whole HDFS filesystem.
> So looks like there is a dependency on amount of memory used by `hadoop fs
> -setrep` command on how many files total HDFS has? This is not a huge HDFS
> filesystem, I would say even "small" by current standards.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[VOTE] Merge YARN-3926 (resource profile) to trunk
Hi folks, Per earlier discussion [1], I'd like to start a formal vote to merge feature branch YARN-3926 (Resource profile) to trunk. The vote will run for 7 days and will end August 30 10:00 AM PDT. Briefly, YARN-3926 can extend resource model of YARN to support resource types other than CPU and memory, so it will be a cornerstone of features like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA support (YARN-5983), network IO scheduling/isolation (YARN-2140). In addition to that, YARN-3926 allows admin to preconfigure resource profiles in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB disk>, so applications can request "m3.large" profile instead of specifying all resource types’s values. There are 32 subtasks that were completed as part of this effort. This feature needs to be explicitly turned on before use. We paid close attention to compatibility, performance, and scalability of this feature, mentioned in [1], we didn't see observable performance regression in large scale SLS (scheduler load simulator) executions and saw less than 5% performance regression by using micro benchmark added by YARN-6775. This feature works from end-to-end (including UI/CLI/application/server), we have setup a cluster with this feature turned on runs for several weeks, we didn't see any issues by far. Merge JIRA: YARN-7013 (Jenkins gave +1 already). Documentation: YARN-7056 Special thanks to a team of folks who worked hard and contributed towards this effort including design discussion/development/reviews, etc.: Varun Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, Karthik Kambatla, Jason Lowe, Arun Suresh. Regards, Wangda Tan [1] http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%3DBbp4G8inQZmaMg%40mail.gmail.com%3E
[jira] [Created] (HDFS-12346) [branch-2] Combine the old and the new chooseRandom for better performance
Chen Liang created HDFS-12346: - Summary: [branch-2] Combine the old and the new chooseRandom for better performance Key: HDFS-12346 URL: https://issues.apache.org/jira/browse/HDFS-12346 Project: Hadoop HDFS Issue Type: Improvement Reporter: Chen Liang Assignee: Chen Liang This JIRA is to backport HDFS-11577 back to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12341) Ozone: maven dist compilation fails with "Duplicate classes found" error
Mukul Kumar Singh created HDFS-12341:
Summary: Ozone: maven dist compilation fails with "Duplicate
classes found" error
Key: HDFS-12341
URL: https://issues.apache.org/jira/browse/HDFS-12341
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
{{mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true}} fails with
the following error
{code}
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-beta1-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-beta1-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/io/netty/buffer/ByteBufProcessor$3.class
org/apache/hadoop/shaded/io/netty/handler/codec/sctp/SctpOutboundByteStreamHandler.class
org/apache/hadoop/shaded/io/netty/handler/codec/rtsp/RtspObjectEncoder.class
org/apache/hadoop/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe.class
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
