Re: [EVENT] HDFS Bug Bash: March 12
I cannot join this in person as I am currently oversea. but +1 on this event. Thanks for organizing this, Chris! Please let me know if anything else I can help here. Thanks, Junping 2018-03-05 16:03 GMT-08:00 Chris Douglas: > [Cross-posting, as this affects the rest of the project] > > Hey folks- > > As discussed last month [1], the HDFS build hasn't been healthy > recently. We're dedicating a bug bash to stabilize the build and > address some longstanding issues with our unit tests. We rely on our > CI infrastructure to keep the project releasable, and in its current > state, it's not protecting us from regressions. While we probably > won't achieve all our goals in this session, we can develop the > conditions for reestablishing a firm foundation. > > If you're new to the project, please consider attending and > contributing. Committers often prioritize large or complicated > patches, and the issues that make the project livable don't get enough > attention. A bug bash is a great opportunity to pull reviewers' > collars, and fix the annoyances that slow us all down. > > If you're a committer, please join us! While some of the proposed > repairs are rote, many unit tests rely on implementation details and > non-obvious invariants. We need domain experts to help untangle > complex dependencies and to prevent breakage of deliberate, but > counter-intuitive code. > > We're collecting tasks in wiki [2] and will include a dial-in option > for folks who aren't local. > > Meetup has started charging for creating new events, so we'll have to > find another way to get an approximate headcount and publish the > address. Please ping me if you have a preferred alternative. -C > > [1]: https://s.apache.org/nEoQ > [2]: https://cwiki.apache.org/confluence/pages/viewpage. > action?pageId=75965105 > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Re: [VOTE] Merging branch HDFS-7240 to trunk
Hi Joep, On Tue, Mar 6, 2018 at 6:50 PM, J. Rottinghuiswrote: Obviously when people do want to use Ozone, then having it in the same repo > is easier. The flipside is that, separate top-level project in the same > repo or not, it adds to the Hadoop releases. > Apache projects are about the group of people who are working together. There is a large overlap between the team working on HDFS and Ozone, which is a lot of the motivation to keep project overhead to a minimum and not start a new project. Using the same releases or separate releases is a distinct choice. Many Apache projects, such as Common and Maven, have multiple artifacts that release independently. In Hive, we have two sub-projects that release indepdendently: Hive Storage API, and Hive. One thing we did during that split to minimize the challenges to the developers was that Storage API and Hive have the same master branch. However, since they have different releases, they have their own release branches and release numbers. If there is a change in Ozone and a new release needed, it would have to > wait for a Hadoop release. Ditto if there is a Hadoop release and there is > an issue with Ozone. The case that one could turn off Ozone through a Maven > profile works only to some extend. > If we have done a 3.x release with Ozone in it, would it make sense to do > a 3.y release with y>x without Ozone in it? That would be weird. > Actually, if Ozone is marked as unstable/evolving (we should actually have an even stronger warning for a feature preview), we could remove it in a 3.x. If a user picks up a feature before it is stable, we try to provide a stable platform, but mistakes happen. Introducing an incompatible change to the Ozone API between 3.1 and 3.2 wouldn't be good, but it wouldn't be the end of the world. .. Owen
Re: [VOTE] Merging branch HDFS-7240 to trunk
Joep, You raise a number of points: (1) Ozone vs and object stores. “Some users would choose Ozone as that layer, some might use S3, others GCS, or Azure, or something else”. (2) How HDSL/Ozone fits into Hadoop and whether it is necessary. (3) You raise the release issue which we will respond in a separate email. Let me respond to 1 & 2: ***Wrt to (1) Ozone vs other object stores*** Neither HDFS or Ozone has any real role in cloud except for temp data. The cost of local disk or EBS is so high that long term data storage on HDFS or even Ozone is prohibitive. So why the hell create the KV namespace? We need to stabilize the HDSL where data is stored. - We are targeting Hive and SPark apps to stabilize HDSL using real Hadoop apps over OzoneFS. But HDSL/Ozone is not feature compatible with HDFS so how will users even use it for real to stability. Users can run HDFS and Ozone side by side in same cluster and have two namespace (just like in Federation) and run apps on both: run some hive and spark apps on Ozone and others that need full HDFS feature (encryption) on HDFS. As it becomes stable they can start using HDSL/Ozone for production use for a portion of their data. ***Wrt to (2) HDSL/Ozone fitting into Hadoop and why the same repository*** Ozone KV is a temporary step. Real goal is to put NN on top of HDSL, We have shown how to do that in the roadmap that Konstantine and Chris D asked. Milestone 1 is feasible and doesn't require removal of FSN lock. We have also shown several cases of sharing other code in future (protocol engine). This co-development will be easier if in the same repo. Over time HDSL + ported NN will create a new HDFS and become feature compatible - some of the feature will come for free because they are in NN and will port over to the new NN, Some are in block layer (erasure code) and will have to be added to HDSL. --- You compare with Yarn, HDFS and Common. HDFS and Yarn are independent but both depend on Hadoop common (e.g. HBase runs on HDFS without Yarn). HDSL and Ozone will depend on Hadoop common, Indeed the new protocol engine of HDSL might move to Hadoop common or HDFS. We have made sure that there are no dependencies of HDFS on HDSL or currently. ***The Repo issue and conclusion*** HDFS community will need to work together as we evolve old HDFS to use HDSL, new protocol engine and Raft. and together evolve to a newer more powerful set of sub components. It is important that they are in same repo and that we can share code through both private interface. We are not trying to build a competing Object store but to improve HDFS and fixing scalability fundamentally is hard and we are asking for an environment for that to happen easily over the next year while heeding to the stability concerns of HDFS developers (eg we remove compile time dependency, maven profile). This work is not being done by members of foreign project trying to insert code in Hadoop, but by Hadoop/HDFS developers with given track record s and are active participation in Hadoop and HDFS. Our jobs depend on HDFS/Hadoop stability - destabilizing is the last thing we want to do; we have responded every constructive feedback sanjay > On Mar 6, 2018, at 6:50 PM, J. Rottinghuiswrote: > > Sorry for jumping in late into the fray of this discussion. > > It seems Ozone is a large feature. I appreciate the development effort and > the desire to get this into the hands of users. > I understand the need to iterate quickly and to reduce overhead for > development. > I also agree that Hadoop can benefit from a quicker release cycle. For our > part, this is a challenge as we have a large installation with multiple > clusters and thousands of users. It is a constant balance between jumping > to the newest release and the cost of this integration and test at our > scale, especially when things aren't backwards compatible. We try to be > good citizens and upstream our changes and contribute back. > > The point was made that splitting the projects such as common and Yarn > didn't work and had to be reverted. That was painful and a lot of work for > those involved for sure. This project may be slightly different in that > hadoop-common, Yarn and HDFS made for one consistent whole. One couldn't > run a project without the other. > > Having a separate block management layer with possibly multiple block > implementation as pluggable under the covers would be a good future > development for HDFS. Some users would choose Ozone as that layer, some > might use S3, others GCS, or Azure, or something else. > If the argument is made that nobody will be able to run Hadoop as a > consistent stack without Ozone, then that would be a strong case to keep > things in the same repo. > > Obviously when people do want to use Ozone, then having it in the same repo > is easier. The flipside is that, separate top-level project in the same > repo or not, it adds to the Hadoop
Re: [EVENT] HDFS Bug Bash: March 12
+1 Thanks for making this happens! Tsz-Wo On Tuesday, March 6, 2018, 7:49:06 PM PST, Chris Douglaswrote: Found a meetup alternative (thanks Subru): https://meetingstar.io/event/fk13172f1d75KN So we can get a rough headcount, please add (local) if you plan to attend in-person. -C On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas wrote: > [Cross-posting, as this affects the rest of the project] > > Hey folks- > > As discussed last month [1], the HDFS build hasn't been healthy > recently. We're dedicating a bug bash to stabilize the build and > address some longstanding issues with our unit tests. We rely on our > CI infrastructure to keep the project releasable, and in its current > state, it's not protecting us from regressions. While we probably > won't achieve all our goals in this session, we can develop the > conditions for reestablishing a firm foundation. > > If you're new to the project, please consider attending and > contributing. Committers often prioritize large or complicated > patches, and the issues that make the project livable don't get enough > attention. A bug bash is a great opportunity to pull reviewers' > collars, and fix the annoyances that slow us all down. > > If you're a committer, please join us! While some of the proposed > repairs are rote, many unit tests rely on implementation details and > non-obvious invariants. We need domain experts to help untangle > complex dependencies and to prevent breakage of deliberate, but > counter-intuitive code. > > We're collecting tasks in wiki [2] and will include a dial-in option > for folks who aren't local. > > Meetup has started charging for creating new events, so we'll have to > find another way to get an approximate headcount and publish the > address. Please ping me if you have a preferred alternative. -C > > [1]: https://s.apache.org/nEoQ > [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105 - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [EVENT] HDFS Bug Bash: March 12
Hey folks- Remember we're getting together on Monday to fix some "quality of life" issues in the build. Please RSVP [1] so we can preprint badges, that sort of thing. The meetup site only allows for 6 hour events, but we have the room through 6:00PM PST and can keep the bridge/channel open for other timezones that want to keep going. We'll post remote access information when we get it. Please update the wiki [2] as you think of issues we should address. -C [1] https://meetingstar.io/event/fk13172f1d75KN [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105 On Tue, Mar 6, 2018 at 7:48 PM, Chris Douglaswrote: > Found a meetup alternative (thanks Subru): > https://meetingstar.io/event/fk13172f1d75KN > > So we can get a rough headcount, please add (local) if you plan to > attend in-person. -C > > > On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas wrote: >> [Cross-posting, as this affects the rest of the project] >> >> Hey folks- >> >> As discussed last month [1], the HDFS build hasn't been healthy >> recently. We're dedicating a bug bash to stabilize the build and >> address some longstanding issues with our unit tests. We rely on our >> CI infrastructure to keep the project releasable, and in its current >> state, it's not protecting us from regressions. While we probably >> won't achieve all our goals in this session, we can develop the >> conditions for reestablishing a firm foundation. >> >> If you're new to the project, please consider attending and >> contributing. Committers often prioritize large or complicated >> patches, and the issues that make the project livable don't get enough >> attention. A bug bash is a great opportunity to pull reviewers' >> collars, and fix the annoyances that slow us all down. >> >> If you're a committer, please join us! While some of the proposed >> repairs are rote, many unit tests rely on implementation details and >> non-obvious invariants. We need domain experts to help untangle >> complex dependencies and to prevent breakage of deliberate, but >> counter-intuitive code. >> >> We're collecting tasks in wiki [2] and will include a dial-in option >> for folks who aren't local. >> >> Meetup has started charging for creating new events, so we'll have to >> find another way to get an approximate headcount and publish the >> address. Please ping me if you have a preferred alternative. -C >> >> [1]: https://s.apache.org/nEoQ >> [2]: >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105 - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/ [Mar 8, 2018 8:02:34 AM] (yqlin) HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by [Mar 8, 2018 10:13:36 AM] (wwei) YARN-8011. [Mar 8, 2018 11:15:46 AM] (stevel) HADOOP-15292. Distcp's use of pread is slowing it down. Contributed by [Mar 8, 2018 11:24:06 AM] (stevel) HADOOP-15273.distcp can't handle remote stores with different checksum [Mar 8, 2018 5:32:05 PM] (inigoiri) HDFS-13232. RBF: ConnectionManager's cleanup task will compare each [Mar 8, 2018 6:17:02 PM] (xiao) HADOOP-15280. TestKMS.testWebHDFSProxyUserKerb and [Mar 8, 2018 6:23:36 PM] (sunilg) YARN-7944. [UI2] Remove master node link from headers of application -1 overall The following subsystems voted -1: findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources At Resource.java:by returning Resource.resources At Resource.java:[line 234] Failed junit tests : hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-compile-javac-root.txt [296K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/whitespace-eol.txt [9.2M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/whitespace-tabs.txt [288K] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [320K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/716/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org