[jira] [Created] (HADOOP-16598) Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches
Duo Zhang created HADOOP-16598: -- Summary: Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches Key: HADOOP-16598 URL: https://issues.apache.org/jira/browse/HADOOP-16598 Project: Hadoop Common Issue Type: Sub-task Components: common Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
VOTE PASSED - Hadoop-3.1.3-RC0 Re: [VOTE] Release Hadoop-3.1.3-RC0
Hi, Thank you all who spent time to verify and vote 3.1.3 release! I‘m pleased to announce that the vote for 3.1.3 RC0 passed! I'll push the release bits and send out an announcement for 3.1.3 soon. Summary of votes for Hadoop-3.1.3-RC0: 4 binding +1s, from Rohith Sharma K S, Sunil Govindan, Weiwei Yang, Eric Payne 3 non-binding +1s, from Xun Liu, Zac Zhou, Zhankun Tang 1 non-binding with +0s from Masatake Iwasaki and *no -1s*. BR, Zhankun On Tue, 24 Sep 2019 at 15:40, zac yuan wrote: > Looking forward to 3.1.3 release. Thanks Zhankun for your great effort~ > > +1 (non-binding) > > Zac Zhou > > Xun Liu 于2019年9月23日周一 上午11:49写道： > > > +1 > > > > Thanks Zhankun for all of your hard work on this release. > > > > > > > On Sep 20, 2019, at 5:34 AM, epa...@apache.org wrote: > > > > > > > > > > > > +1 (binding) > > > > > > Thanks Zhankun for all of your hard work on this release. > > > > > > I downloaded and built the source and ran it on an insecure multi-node > > pseudo cluster. > > > > > > I performed various YARN manual tests, including creating custom > > resources, creating queue submission ACLs, and queue refreshes. > > > > > > One concern is that preemption does not seem to be working when only > the > > custom resources are over the queue capacity, but I don't think this is > > something introduced with this release. > > > > > > -Eric > > > > > > > > > > > > On Thursday, September 12, 2019, 3:04:44 AM CDT, Zhankun Tang < > > zt...@apache.org> wrote: > > > > > > > > > > > > > > > > > > Hi folks, > > > > > > Thanks to everyone's help on this release. Special thanks to Rohith, > > > Wei-Chiu, Akira, Sunil, Wangda! > > > > > > I have created a release candidate (RC0) for Apache Hadoop 3.1.3. > > > > > > The RC release artifacts are available at: > > > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/ > > > > > > The maven artifacts are staged at: > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1228/ > > > > > > The RC tag in git is here: > > > https://github.com/apache/hadoop/tree/release-3.1.3-RC0 > > > > > > And my public key is at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.* > > > > > > For the testing, I have run several Spark and distributed shell jobs in > > my > > > pseudo cluster. > > > > > > My +1 (non-binding) to start. > > > > > > BR, > > > Zhankun > > > > > > On Wed, 4 Sep 2019 at 15:56, zhankun tang > wrote: > > > > > >> Hi all, > > >> > > >> Thanks for everyone helping in resolving all the blockers targeting > > Hadoop > > >> 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers > > issues > > >> to 3.1.4. > > >> > > >> I'll cut the branch today and call a release vote soon. Thanks! > > >> > > >> > > >> [1]. https://s.apache.org/5hj5i > > >> > > >> BR, > > >> Zhankun > > >> > > >> > > >> On Wed, 21 Aug 2019 at 12:38, Zhankun Tang wrote: > > >> > > >>> Hi folks, > > >>> > > >>> We have Apache Hadoop 3.1.2 released on Feb 2019. > > >>> > > >>> It's been more than 6 months passed and there're > > >>> > > >>> 246 fixes[1]. 2 blocker and 4 critical Issues [2] > > >>> > > >>> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker) > > >>> > > >>> > > >>> I propose my plan to do a maintenance release of 3.1.3 in the next > few > > >>> (one or two) weeks. > > >>> > > >>> Hadoop 3.1.3 release plan: > > >>> > > >>> Code Freezing Date: *25th August 2019 PDT* > > >>> > > >>> Release Date: *31th August 2019 PDT* > > >>> > > >>> > > >>> Please feel free to share your insights on this. Thanks! > > >>> > > >>> > > >>> [1] https://s.apache.org/zw8l5 > > >>> > > >>> [2] https://s.apache.org/fjol5 > > >>> > > >>> > > >>> BR, > > >>> > > >>> Zhankun > > >>> > > >> > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > - > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > >
[jira] [Created] (HADOOP-16597) Backport the protobuf maven plugin change to all active branches
Duo Zhang created HADOOP-16597: -- Summary: Backport the protobuf maven plugin change to all active branches Key: HADOOP-16597 URL: https://issues.apache.org/jira/browse/HADOOP-16597 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Duo Zhang With protobuf-maven-plugin, we do not need to install protoc on the build machine any more, which is really good for developpers. So I think this should go into all active branches. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16599) Allow a SignerInitializer to be specified along with a Custom Signer
Siddharth Seth created HADOOP-16599: --- Summary: Allow a SignerInitializer to be specified along with a Custom Signer Key: HADOOP-16599 URL: https://issues.apache.org/jira/browse/HADOOP-16599 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Environment: A Reporter: Siddharth Seth Assignee: Siddharth Seth HADOOP-16445 added support for custom signers. This is a follow up to allow for an Initializer to be specified along with the Custom Signer, for any initialization etc that is required by the custom signer specified. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[ https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun resolved HADOOP-16600. -- Resolution: Duplicate > StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 > - > > Key: HADOOP-16600 > URL: https://issues.apache.org/jira/browse/HADOOP-16600 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1, 3.1.2 >Reporter: Lisheng Sun >Priority: Major > > details see HADOOP-15398 > Problem: hadoop trunk compilation is failing > Root Cause: > compilation error is coming from > org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is > "The method getArgumentAt(int, Class) is undefined for the > type InvocationOnMock". > StagingTestBase is using getArgumentAt(int, Class) method > which is not available in mockito-all 1.8.5 version. getArgumentAt(int, > Class) method is available only from version 2.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
Lisheng Sun created HADOOP-16600: Summary: StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 Key: HADOOP-16600 URL: https://issues.apache.org/jira/browse/HADOOP-16600 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.1.2, 3.1.1, 3.1.0 Reporter: Lisheng Sun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16600) StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
[ https://issues.apache.org/jira/browse/HADOOP-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun reopened HADOOP-16600: -- > StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 > - > > Key: HADOOP-16600 > URL: https://issues.apache.org/jira/browse/HADOOP-16600 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1, 3.1.2 >Reporter: Lisheng Sun >Priority: Major > > details see HADOOP-15398 > Problem: hadoop trunk compilation is failing > Root Cause: > compilation error is coming from > org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase. Compilation error is > "The method getArgumentAt(int, Class) is undefined for the > type InvocationOnMock". > StagingTestBase is using getArgumentAt(int, Class) method > which is not available in mockito-all 1.8.5 version. getArgumentAt(int, > Class) method is available only from version 2.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
MacChen01 created HADOOP-16601: -- Summary: Add support for hardware crc32 of nativetask checksums on aarch64 arch Key: HADOOP-16601 URL: https://issues.apache.org/jira/browse/HADOOP-16601 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.2.1 Reporter: MacChen01 Fix For: site Add support for aarch64 CRC instructions in nativetask module, optimize the CRC32 and CRC32C. Use the benchmark tools : nttest , the improvement is quite substantial: |ChecksumType|KeyValueType|IO|Before(M/s)|After(M/s)|Improvement| |CRC32|TextType|Write|425.98|602.92|+42%| |Read|796.06|1716.59|+116%| |BytesType|Write|474.25|686.84|+45%| |Read|844.96|1955.03|+131%| |UnknownType|Write|434.84|608.81|+40%| |Read|805.76|1733.82|+115%| |CRC32C|TextType|Write|423.39|606.55|+43%| |Read|799.20|1783.28|+123%| |BytesType|Write|473.95|696.47|+47%| |Read|846.30|2018.06|+138%| |UnknownType|Write|434.07|612.31|+41%| |Read|807.16|1783.95|+121%| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[NEED HELP] Hadoop 3.x Production Deployment Can be Publicly Talk About?
Hi devs and users, Tomorrow (sorry for the short notice) we will do a presentation at Strata Data Conf @ NY for a Community update of Hadoop 3.x. I'm thinking to create a slide about existing production deployment on Hadoop 3.x. Basically, I want to put a logo wall with a list of big names so we can encourage more users to upgrade to 3.x. I knew there are tons of users are on 3.x already, but only a few of them have public slides, I don't get a permission to put other non-public use cases to the slide. So if you are: - Using 3.x in production. (Ideally large scale, using some new features or running in new environment like on cloud, using GPU, etc.). - When I saying 3.x it could be Apache Hadoop 3.x, HDP 3.x, CDH 6.x or any distribution with Apache Hadoop 3.x as base. Please respond your company name w/ logo (it it is not too obvious) to this email (either public or private to me). If you could include a short summary (which can be publicly shared) of the use cases that will be very helpful. If the number of companies responded is too low, I may skip putting a logo wall. Hope to get your feedbacks soon. Thanks, Wangda Tan
Incompatible changes between branch-2.8 and branch-2.9
We (Verizon Media) are currently moving towards upgrading our clusters from
our internal fork of branch-2.8 to an internal fork of branch-2. During
this process, we have found multiple incompatible changes in protobufs
between branch-2.8 and branch-2. These incompatibilities were all
introduced between branch-2.8 and branch-2.9. I did a git diff over all
.proto files across the branch-2.8 and branch-2.9 and found 3 instances of
incompatibilities from 3 separate commits. All of the incompatibilities are
in yarn_protos.proto
I would like to discuss how to fix these incompatible changes. Otherwise,
rolling upgrades will not be supported between branch-2.8 (or below) and
branch-2.9 (or beyond). We could revert the incompatible changes, but then
the new releases would be incompatible with the releases that have these
incompatible changes. If we do nothing, then rolling upgrades won't work
between 2.8- and 2.9+.
Thanks,
Eric
---
git diff branch-2.8..branch-2.9 $(find . -name '*\.proto')
https://issues.apache.org/jira/browse/YARN-6616
- Trunk patch (applied through branch-2.9) differs from branch-2.8 patch
@@ -211,7 +245,20 @@ message ApplicationReportProto {
optional PriorityProto priority = 23;
optional string appNodeLabelExpression = 24;
optional string amNodeLabelExpression = 25;
- optional int64 submitTime = 26;
+ repeated AppTimeoutsMapProto appTimeouts = 26;
+ optional int64 launchTime = 27;
+ optional int64 submitTime = 28;
https://issues.apache.org/jira/browse/YARN-6050
- Trunk and branch-2 patches both change the protobuf type in the same
way.
@@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto {
optional LogAggregationContextProto log_aggregation_context = 14;
optional ReservationIdProto reservation_id = 15;
optional string node_label_expression = 16;
- optional ResourceRequestProto am_container_resource_request = 17;
+ repeated ResourceRequestProto am_container_resource_request = 17;
+ repeated ApplicationTimeoutMapProto application_timeouts = 18;
https://issues.apache.org/jira/browse/YARN-7813
- Trunk (applied through branch-3.1) and branch-3.0 (applied through
branch-2.9) patches differ from branch-2.8 patch
@@ -425,7 +501,21 @@ message QueueInfoProto {
optional string defaultNodeLabelExpression = 9;
optional QueueStatisticsProto queueStatistics = 10;
optional bool preemptionDisabled = 11;
- optional bool intraQueuePreemptionDisabled = 12;
+ repeated QueueConfigurationsMapProto queueConfigurationsMap = 12;
+ optional bool intraQueuePreemptionDisabled = 13;
Re: Incompatible changes between branch-2.8 and branch-2.9
Hi Eric, thanks for the investigation.
- For YARN-6616, for branch-2.8 and below, it was only committed to
2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
revert YARN-6616 from branch-2.7 and branch-2.8.
- For YARN-6050, there's a bit here:
https://developers.google.com/protocol-buffers/docs/proto that says
"optional is compatible with repeated", so I think we should be OK there.
- For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5
to a 2.9+ version will be an issue. One option could be to move the
intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to
release this), then upgrade from 2.8.6 to 2.9+.
Jonathan Hung
On Tue, Sep 24, 2019 at 9:23 AM Eric Badger
wrote:
> We (Verizon Media) are currently moving towards upgrading our clusters from
> our internal fork of branch-2.8 to an internal fork of branch-2. During
> this process, we have found multiple incompatible changes in protobufs
> between branch-2.8 and branch-2. These incompatibilities were all
> introduced between branch-2.8 and branch-2.9. I did a git diff over all
> .proto files across the branch-2.8 and branch-2.9 and found 3 instances of
> incompatibilities from 3 separate commits. All of the incompatibilities are
> in yarn_protos.proto
>
>
> I would like to discuss how to fix these incompatible changes. Otherwise,
> rolling upgrades will not be supported between branch-2.8 (or below) and
> branch-2.9 (or beyond). We could revert the incompatible changes, but then
> the new releases would be incompatible with the releases that have these
> incompatible changes. If we do nothing, then rolling upgrades won't work
> between 2.8- and 2.9+.
>
>
> Thanks,
>
>
> Eric
>
>
> ---
>
>
> git diff branch-2.8..branch-2.9 $(find . -name '*\.proto')
>
>
> https://issues.apache.org/jira/browse/YARN-6616
>
>- Trunk patch (applied through branch-2.9) differs from branch-2.8 patch
>
> @@ -211,7 +245,20 @@ message ApplicationReportProto {
>
>optional PriorityProto priority = 23;
>
>optional string appNodeLabelExpression = 24;
>
>optional string amNodeLabelExpression = 25;
>
> - optional int64 submitTime = 26;
>
> + repeated AppTimeoutsMapProto appTimeouts = 26;
>
> + optional int64 launchTime = 27;
>
> + optional int64 submitTime = 28;
>
>
> https://issues.apache.org/jira/browse/YARN-6050
>
>- Trunk and branch-2 patches both change the protobuf type in the same
>way.
>
> @@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto {
>
>optional LogAggregationContextProto log_aggregation_context = 14;
>
>optional ReservationIdProto reservation_id = 15;
>
>optional string node_label_expression = 16;
>
> - optional ResourceRequestProto am_container_resource_request = 17;
>
> + repeated ResourceRequestProto am_container_resource_request = 17;
>
> + repeated ApplicationTimeoutMapProto application_timeouts = 18;
>
>
> https://issues.apache.org/jira/browse/YARN-7813
>
>- Trunk (applied through branch-3.1) and branch-3.0 (applied through
>branch-2.9) patches differ from branch-2.8 patch
>
> @@ -425,7 +501,21 @@ message QueueInfoProto {
>
>optional string defaultNodeLabelExpression = 9;
>
>optional QueueStatisticsProto queueStatistics = 10;
>
>optional bool preemptionDisabled = 11;
>
> - optional bool intraQueuePreemptionDisabled = 12;
>
> + repeated QueueConfigurationsMapProto queueConfigurationsMap = 12;
>
> + optional bool intraQueuePreemptionDisabled = 13;
>
[jira] [Resolved] (HADOOP-16560) [YARN] use protobuf-maven-plugin to generate protobuf classes
[ https://issues.apache.org/jira/browse/HADOOP-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HADOOP-16560. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Merged already to trunk > [YARN] use protobuf-maven-plugin to generate protobuf classes > - > > Key: HADOOP-16560 > URL: https://issues.apache.org/jira/browse/HADOOP-16560 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Duo Zhang >Priority: Major > Fix For: 3.3.0 > > > Use "protoc-maven-plugin" to dynamically download protobuf executable to > generate protobuf classes from proto file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16561) [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes
[ https://issues.apache.org/jira/browse/HADOOP-16561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HADOOP-16561. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Merged to trunk. > [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes > -- > > Key: HADOOP-16561 > URL: https://issues.apache.org/jira/browse/HADOOP-16561 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Duo Zhang >Priority: Major > Fix For: 3.3.0 > > > Use "protoc-maven-plugin" to dynamically download protobuf executable to > generate protobuf classes from proto file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16595) [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf
Vinayakumar B created HADOOP-16595: -- Summary: [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf Key: HADOOP-16595 URL: https://issues.apache.org/jira/browse/HADOOP-16595 Project: Hadoop Common Issue Type: Sub-task Components: hadoop-thirdparty Reporter: Vinayakumar B Create a separate repo "hadoop-thirdparty" to have shaded dependencies. starting with protobuf-java:3.7.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16596) [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency
Vinayakumar B created HADOOP-16596: -- Summary: [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency Key: HADOOP-16596 URL: https://issues.apache.org/jira/browse/HADOOP-16596 Project: Hadoop Common Issue Type: Sub-task Reporter: Vinayakumar B Use the shaded protobuf classes from "hadoop-thirdparty" in hadoop codebase. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Incompatible changes between branch-2.8 and branch-2.9
* For YARN-6616, for branch-2.8 and below, it was only committed to
2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
revert YARN-6616 from branch-2.7 and branch-2.8.
- This seems reasonable. Since we haven't released anything, it should be
no issue to change the 2.7/2.8 protobuf field to have the same value as 2.9+
* For YARN-6050, there's a bit here:
https://developers.google.com/protocol-buffers/docs/proto that says
"optional is compatible with repeated", so I think we should be OK there.
- Optional is compatible with repeatable over the wire such that protobuf
won't blow up, but does that actually mean that it's compatible in this
case? If it's expecting an optional and gets a repeated, it's going to drop
everything except for the last value. I don't know enough about YARN-6050
to say if this will be ok or not.
* For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5
to a 2.9+ version will be an issue. One option could be to move the
intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to
release this), then upgrade from 2.8.6 to 2.9+.
- I'm ok with this, but it should be noted that the upgrade from
2.8.4/2.8.5 to 2.8.6 (or 2.9+) would not be compatible for a rolling
upgrade. So this would cause some pain to anybody with clusters on those
versions.
Eric
On Tue, Sep 24, 2019 at 2:42 PM Jonathan Hung wrote:
> Sorry, let me edit my first point. We can just create addendums for
> YARN-6616 in branch-2.7 and branch-2.8 to edit the submitTime field to the
> correct id 28. We don’t need to revert YARN-6616 from these branches
> completely.
>
> Jonathan
>
>
> From: Jonathan Hung
> Sent: Tuesday, September 24, 2019 11:38 AM
> To: Eric Badger
> Cc: Hadoop Common; yarn-dev; mapreduce-dev; Hdfs-dev
> Subject: Re: Incompatible changes between branch-2.8 and branch-2.9
>
> Hi Eric, thanks for the investigation.
>
> * For YARN-6616, for branch-2.8 and below, it was only committed to
> 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
> revert YARN-6616 from branch-2.7 and branch-2.8.
> * For YARN-6050, there's a bit here:
> https://developers.google.com/protocol-buffers/docs/proto that says
> "optional is compatible with repeated", so I think we should be OK there.
> * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or
> 2.8.5 to a 2.9+ version will be an issue. One option could be to move the
> intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
> users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to
> release this), then upgrade from 2.8.6 to 2.9+.
>
> Jonathan Hung
>
>
> On Tue, Sep 24, 2019 at 9:23 AM Eric Badger
> wrote:
> We (Verizon Media) are currently moving towards upgrading our clusters from
> our internal fork of branch-2.8 to an internal fork of branch-2. During
> this process, we have found multiple incompatible changes in protobufs
> between branch-2.8 and branch-2. These incompatibilities were all
> introduced between branch-2.8 and branch-2.9. I did a git diff over all
> .proto files across the branch-2.8 and branch-2.9 and found 3 instances of
> incompatibilities from 3 separate commits. All of the incompatibilities are
> in yarn_protos.proto
>
>
> I would like to discuss how to fix these incompatible changes. Otherwise,
> rolling upgrades will not be supported between branch-2.8 (or below) and
> branch-2.9 (or beyond). We could revert the incompatible changes, but then
> the new releases would be incompatible with the releases that have these
> incompatible changes. If we do nothing, then rolling upgrades won't work
> between 2.8- and 2.9+.
>
>
> Thanks,
>
>
> Eric
>
>
> ---
>
>
> git diff branch-2.8..branch-2.9 $(find . -name '*\.proto')
>
>
> https://issues.apache.org/jira/browse/YARN-6616
>
>- Trunk patch (applied through branch-2.9) differs from branch-2.8 patch
>
> @@ -211,7 +245,20 @@ message ApplicationReportProto {
>
>optional PriorityProto priority = 23;
>
>optional string appNodeLabelExpression = 24;
>
>optional string amNodeLabelExpression = 25;
>
> - optional int64 submitTime = 26;
>
> + repeated AppTimeoutsMapProto appTimeouts = 26;
>
> + optional int64 launchTime = 27;
>
> + optional int64 submitTime = 28;
>
>
> https://issues.apache.org/jira/browse/YARN-6050
>
>- Trunk and branch-2 patches both change the protobuf type in the same
>way.
>
> @@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto {
>
>optional LogAggregationContextProto log_aggregation_context = 14;
>
>optional ReservationIdProto reservation_id = 15;
>
>optional string node_label_expression = 16;
>
> - optional ResourceRequestProto am_container_resource_request = 17;
>
> + repeated ResourceRequestProto am_container_resource_request = 17;
>
Re: Incompatible changes between branch-2.8 and branch-2.9
Sorry, let me edit my first point. We can just create addendums for YARN-6616
in branch-2.7 and branch-2.8 to edit the submitTime field to the correct id 28.
We don’t need to revert YARN-6616 from these branches completely.
Jonathan
From: Jonathan Hung
Sent: Tuesday, September 24, 2019 11:38 AM
To: Eric Badger
Cc: Hadoop Common; yarn-dev; mapreduce-dev; Hdfs-dev
Subject: Re: Incompatible changes between branch-2.8 and branch-2.9
Hi Eric, thanks for the investigation.
* For YARN-6616, for branch-2.8 and below, it was only committed to
2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
revert YARN-6616 from branch-2.7 and branch-2.8.
* For YARN-6050, there's a bit here:
https://developers.google.com/protocol-buffers/docs/proto that says "optional
is compatible with repeated", so I think we should be OK there.
* For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5 to
a 2.9+ version will be an issue. One option could be to move the
intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to release
this), then upgrade from 2.8.6 to 2.9+.
Jonathan Hung
On Tue, Sep 24, 2019 at 9:23 AM Eric Badger
wrote:
We (Verizon Media) are currently moving towards upgrading our clusters from
our internal fork of branch-2.8 to an internal fork of branch-2. During
this process, we have found multiple incompatible changes in protobufs
between branch-2.8 and branch-2. These incompatibilities were all
introduced between branch-2.8 and branch-2.9. I did a git diff over all
.proto files across the branch-2.8 and branch-2.9 and found 3 instances of
incompatibilities from 3 separate commits. All of the incompatibilities are
in yarn_protos.proto
I would like to discuss how to fix these incompatible changes. Otherwise,
rolling upgrades will not be supported between branch-2.8 (or below) and
branch-2.9 (or beyond). We could revert the incompatible changes, but then
the new releases would be incompatible with the releases that have these
incompatible changes. If we do nothing, then rolling upgrades won't work
between 2.8- and 2.9+.
Thanks,
Eric
---
git diff branch-2.8..branch-2.9 $(find . -name '*\.proto')
https://issues.apache.org/jira/browse/YARN-6616
- Trunk patch (applied through branch-2.9) differs from branch-2.8 patch
@@ -211,7 +245,20 @@ message ApplicationReportProto {
optional PriorityProto priority = 23;
optional string appNodeLabelExpression = 24;
optional string amNodeLabelExpression = 25;
- optional int64 submitTime = 26;
+ repeated AppTimeoutsMapProto appTimeouts = 26;
+ optional int64 launchTime = 27;
+ optional int64 submitTime = 28;
https://issues.apache.org/jira/browse/YARN-6050
- Trunk and branch-2 patches both change the protobuf type in the same
way.
@@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto {
optional LogAggregationContextProto log_aggregation_context = 14;
optional ReservationIdProto reservation_id = 15;
optional string node_label_expression = 16;
- optional ResourceRequestProto am_container_resource_request = 17;
+ repeated ResourceRequestProto am_container_resource_request = 17;
+ repeated ApplicationTimeoutMapProto application_timeouts = 18;
https://issues.apache.org/jira/browse/YARN-7813
- Trunk (applied through branch-3.1) and branch-3.0 (applied through
branch-2.9) patches differ from branch-2.8 patch
@@ -425,7 +501,21 @@ message QueueInfoProto {
optional string defaultNodeLabelExpression = 9;
optional QueueStatisticsProto queueStatistics = 10;
optional bool preemptionDisabled = 11;
- optional bool intraQueuePreemptionDisabled = 12;
+ repeated QueueConfigurationsMapProto queueConfigurationsMap = 12;
+ optional bool intraQueuePreemptionDisabled = 13;
Re: Incompatible changes between branch-2.8 and branch-2.9
- I've created YARN-9855 and uploaded patches to fix YARN-6616 in
branch-2.8 and branch-2.7.
- For YARN-6050, not sure either. Robert/Wangda, can you comment on
YARN-6050 compatibility?
- For YARN-7813, not sure why moving from 2.8.4/5 -> 2.8.6 would be
incompatible with this strategy? It should be OK to remove/add optional
fields (removing the field with id 12, and adding the field with id 13).
The difficulties I see here are, we would have to leave id 12 blank in
2.8.6 (so we cannot have YARN-6164 in branch-2.8), and users on 2.8.4/5
would have to move to 2.8.6 before moving to 2.9+. But rolling upgrade
would still work IIUC.
Jonathan Hung
On Tue, Sep 24, 2019 at 2:52 PM Eric Badger
wrote:
> * For YARN-6616, for branch-2.8 and below, it was only committed to
> 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
> revert YARN-6616 from branch-2.7 and branch-2.8.
> - This seems reasonable. Since we haven't released anything, it should
> be no issue to change the 2.7/2.8 protobuf field to have the same value as
> 2.9+
>
> * For YARN-6050, there's a bit here:
> https://developers.google.com/protocol-buffers/docs/proto that says
> "optional is compatible with repeated", so I think we should be OK there.
> - Optional is compatible with repeatable over the wire such that
> protobuf won't blow up, but does that actually mean that it's compatible in
> this case? If it's expecting an optional and gets a repeated, it's going to
> drop everything except for the last value. I don't know enough about
> YARN-6050 to say if this will be ok or not.
>
> * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5
> to a 2.9+ version will be an issue. One option could be to move the
> intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
> users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to
> release this), then upgrade from 2.8.6 to 2.9+.
> - I'm ok with this, but it should be noted that the upgrade from
> 2.8.4/2.8.5 to 2.8.6 (or 2.9+) would not be compatible for a rolling
> upgrade. So this would cause some pain to anybody with clusters on those
> versions.
>
> Eric
>
> On Tue, Sep 24, 2019 at 2:42 PM Jonathan Hung
> wrote:
>
>> Sorry, let me edit my first point. We can just create addendums for
>> YARN-6616 in branch-2.7 and branch-2.8 to edit the submitTime field to the
>> correct id 28. We don’t need to revert YARN-6616 from these branches
>> completely.
>>
>> Jonathan
>>
>>
>> From: Jonathan Hung
>> Sent: Tuesday, September 24, 2019 11:38 AM
>> To: Eric Badger
>> Cc: Hadoop Common; yarn-dev; mapreduce-dev; Hdfs-dev
>> Subject: Re: Incompatible changes between branch-2.8 and branch-2.9
>>
>> Hi Eric, thanks for the investigation.
>>
>> * For YARN-6616, for branch-2.8 and below, it was only committed to
>> 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can
>> revert YARN-6616 from branch-2.7 and branch-2.8.
>> * For YARN-6050, there's a bit here:
>> https://developers.google.com/protocol-buffers/docs/proto that says
>> "optional is compatible with repeated", so I think we should be OK there.
>> * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or
>> 2.8.5 to a 2.9+ version will be an issue. One option could be to move the
>> intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then
>> users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to
>> release this), then upgrade from 2.8.6 to 2.9+.
>>
>> Jonathan Hung
>>
>>
>> On Tue, Sep 24, 2019 at 9:23 AM Eric Badger
>>
>> wrote:
>> We (Verizon Media) are currently moving towards upgrading our clusters
>> from
>> our internal fork of branch-2.8 to an internal fork of branch-2. During
>> this process, we have found multiple incompatible changes in protobufs
>> between branch-2.8 and branch-2. These incompatibilities were all
>> introduced between branch-2.8 and branch-2.9. I did a git diff over all
>> .proto files across the branch-2.8 and branch-2.9 and found 3 instances of
>> incompatibilities from 3 separate commits. All of the incompatibilities
>> are
>> in yarn_protos.proto
>>
>>
>> I would like to discuss how to fix these incompatible changes. Otherwise,
>> rolling upgrades will not be supported between branch-2.8 (or below) and
>> branch-2.9 (or beyond). We could revert the incompatible changes, but then
>> the new releases would be incompatible with the releases that have these
>> incompatible changes. If we do nothing, then rolling upgrades won't work
>> between 2.8- and 2.9+.
>>
>>
>> Thanks,
>>
>>
>> Eric
>>
>>
>> ---
>>
>>
>> git diff branch-2.8..branch-2.9 $(find . -name '*\.proto')
>>
>>
>> https://issues.apache.org/jira/browse/YARN-6616
>>
>>- Trunk patch (applied through branch-2.9) differs from branch-2.8
>> patch
>>
>> @@ -211,7 +245,20 @@ message ApplicationReportProto {
>>
>>
