[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213458#comment-16213458 ] Ray Chiang commented on YARN-6142: -- Minor issues found by JACC. YARN-2696 - CapacityScheduler#getQueueComparator() split into partitioned/nonparititoned comparator YARN-3139 - Removed synchronized from CapacityScheduler#getContainerTokenSecretManager() - Removed synchronized from CapacityScheduler#getRMContext() - Removed synchronized from CapacityScheduler#setRMContext() YARN-3413 - YarnClient#getClusterNodeLabels() changed return type YARN-3866 - Major refactor in Public APIs for AM-RM for handling container resizing. - Change went into both 2.8.0 and 3.0.0. YARN-3873 - CapacityScheduler#getApplicationComparator() removed YARN-4593 - AbstractService#getConfig() removed synchronized YARN-5077 - Removed SchedulingPolicy#checkIfAMResourceUsageOverLimit() YARN-5221 - AllocateRequest / AllocateResponse has methods changed from Public/Stable to Public/Unstable YARN-5713 - Update jackson affects TimelineUtils#dumpTimelineRecordtoJSON() > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213251#comment-16213251 ] Ray Chiang commented on YARN-6142: -- I'm done with the JACC analysis, but need to do the same type of writeup that was done for protobuf. The quick answer is that we don't have any major red flags, but I'm going to note some potential incompatibilities that are very minor, but could affect some random API user out there. > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213177#comment-16213177 ] Andrew Wang commented on YARN-6142: --- Hi Ray, what is left to do here? Is it tracking towards completion by the end of the month? > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175664#comment-16175664 ] Ray Chiang commented on YARN-6142: -- Quick summary. Filed YARN-7219 for the only follow up issue in investigating protobuf issues. > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175663#comment-16175663 ] Ray Chiang commented on YARN-6142: -- Security.proto * Two new messages yarn_protos.proto * ResourceUtilizationProto ** YARN-4293. 2.8.0 and later. * ContainerStateProto added 2 new enum ** YARN-4597. 2.8.0 and later. * ContainerProto added 3 optional fields ** execution_type *** YARN-5127. 2.9.0 and later. ** allocation_request_id *** YARN-4887. 2.9.0 and later. ** version *** YARN-5221. 2.8.0 and later. * FinalApplicationStatusProto added 1 new enum ** YARN-4207. 2.8.0 and later. * ApplicationResourceUsageReportProto added 4 optional fields ** queue_usage_percentage *** YARN-4285. 2.8.0 and later. ** cluster_usage_percentage *** YARN-4285. 2.8.0 and later. ** preempted_memory_seconds *** YARN-4218. 2.8.0 and later. ** preempted_vcore_seconds *** YARN-4218. 2.8.0 and later. * ApplicationReportProto added 6 optional fields ** log_aggregation_status *** YARN-1402. 2.8.0 and later. ** unmanaged_application *** YARN-3543. 2.8.0 and later. ** priority *** YARN-3948. 2.8.0 and later. ** appNodeLabelExpression *** YARN-3717. 2.8.0 and later. ** amNodeLabelExpression *** YARN-3717. 2.8.0 and later. ** appTimeouts *** YARN-5965. 2.9.0 and later. * New message AppTimeoutsMapProto ** YARN-5965. 2.9.0 and later. * New message ApplicationTimeoutProto ** YARN-5965. 2.9.0 and later. * New enum LogAggregationStatusProto ** YARN-1402. 2.8.0 and later. * ApplicationAttemptReportProto added 2 optional fields ** start_time *** YARN-3451. 2.8.0 and later. ** finish_time *** YARN-3451. 2.8.0 and later. * NodeStateProto added 2 new enums ** NS_DECOMMISSIONING *** YARN-3225. 2.8.0 and later. ** NS_SHUTDOWN *** YARN-41. 2.8.0 and later. * NodeReportProto added 2 optional fields ** containers_utilization *** YARN-4293. 2.8.0 and later. ** node_utilization *** YARN-4293. 2.8.0 and later. * New message NodeLabelProto * New enum ContainerTypeProto * New enum ExecutionTypeProto * ResourceRequestProto added 2 optional fields ** execution_type_request *** YARN-5180. 2.9.0 and later. ** allocation_request_id *** YARN-4888. 2.9.0 and later. * New message ExecutionTypeRequestProto ** YARN-4888. 2.9.0 and later. * ApplicationSubmissionContextProto changed 1 field to repeated, added 1 optional field ** am_container_resource_request *** YARN-6050. 2.9.0 and later. ** application_timeouts *** YARN-4205. 2.9.0 and later. * New enum ApplicationTimeoutTypeProto ** YARN-4205. 2.9.0 and later. * New message ApplicationTimeoutMapProto ** YARN-4205. 2.9.0 and later. * New message ApplicationUpdateTimeoutMapProto ** YARN-4205. 2.9.0 and later. * LogAggregationContextProto added 2 optional fields ** log_aggregation_policy_class_name *** YARN-221. 2.8.0 and later. ** log_aggregation_policy_parameters *** YARN-221. 2.8.0 and later. * YarnClusterMetricsProto added 5 optional fields ** All fields YARN-3348 * QueueStateProto added 1 new enum ** YARN-5756. 2.9.0 and later. * New message QueueStatisticsProto ** YARN-3348. 2.8.0 and later. * QueueInfoProto added 3 optional fields ** queueStatistics ** preemptionDisabled ** queueConfigurationsMap * New message QueueConfigurationsProto ** YARN-6164. 2.9.0 and later. * New message QueueConfigurationsMapRoto ** YARN-6164. 2.9.0 and later. * New enum SignalContainerCommandProto ** YARN-1897. 2.8.0 and later. * ReservationDefinitionProto added 2 optional fields ** recurrence_expression *** YARN-5327. 2.9.0 and later. ** priority *** YARN-5384. 2.9.0 and later. * New message ResourceAllocationRequestProto ** YARN-4340. 2.8.0 and later. * New message ReservationAllocationStateProto ** YARN-4340. 2.8.0 and later. * ContainerLaunchContextProto added 2 optional fields ** container_retry_context *** YARN-3998. 2.9.0 and later. ** tokens_conf *** YARN-5910. 2.9.0 and later. * ContainerStatusProto added 3 new optional fields ** capability *** YARN-3866. 2.8.0 and later. ** executionType *** YARN-2882. 2.9.0 and later. ** container_attributes *** YARN-5430. 2.9.0 and later. * Message ContainerResourceIncreaseRequestProto moved to yarn_service_protos.proto ** YARN-3866. 2.8.0 and later. *** Still in 2.8.x * Message ContainerResourceIncreaseProto moved to yarn_service_protos.proto ** YARN-3866. 2.8.0 and later. *** Still in 2.8.x * Message ContainerResourceDecreaseProto moved to yarn_service_protos.proto ** YARN-3866. 2.8.0 and later. *** Still in 2.8.x * New message ContainerRetryContextProto ** YARN-3998. 2.9.0 and later. yarn_server_common_service_protos.proto * New message RemoteNodeProto * New message RegisterDistributedSchedulingAMResponseProto * New message DistributedSchedulingAllocateResponseProto * New message DistributedSchedulingAllocateRequestProto * New message NodeLabelsProto * RegisterNodeManagerRequestProto added 2 optional fields * RegisterNodeManagerResponseProto
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170859#comment-16170859 ] Ray Chiang commented on YARN-6142: -- [~andrew.wang], I'll have to do the rest of the .proto files post beta1, but before GA. For JACC output, I'm down to 217 issues out of the 3207 generated by JACC. All the issues I've found so far are already documented as "Incompatible" in existing JIRAs. > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166655#comment-16166655 ] Ray Chiang commented on YARN-6142: -- h2. Proto Changes h3. ResourceTracker.proto * Added unRegisterNodeManager. Change due to YARN-41. Confirmed in 2.8.0 and later. ** Potential issue with upgraded NM but not RM. => RM should be upgraded first. h3. RPCHeader.proto * Not related to YARN. Looks like support for HTrace. h3. applicationclient_protocol.proto * Added failApplicationAttempt ** YARN-261. Confirmed in 2.8.0 and later. * Added getNewReservation ** YARN-4957. Confirmed in 2.8.0 and later. * Added listReservations ** YARN-4340. Confirmed in 2.8.0 and later. * Added updateApplicationPriority ** YARN-4014. Confirmed in 2.8.0 and later. * Added signalToContainer ** MAPREDUCE-5044. Confirmed in 2.8.0 and later. * Added updateApplicationTimeouts ** YARN-5611. Confirmed in 2.9.0 and later. h3. collectornodemanager_protocol.proto * Requires Timeline Service to be running first? h3. containermanagement_protocol.proto * Added increaseContainersResource ** YARN-1449. 2.8.0 and later * Added updateContainer ** YARN-5977. 2.9.0 and later * signalToContainer ** MAPREDUCE-5044. Confirmed 2.8.0 and later. * Added localizeContainer ** YARN-5557. 2.9.0 and later * Added reInitializeContainer/restartContainer/rollbackContainer/commitContainer ** YARN-5609. 2.9.0 and later h3. distributed_scheduling_am_protocol.proto * All for distributed scheduling support in the AM h3. mr_protos.proto * Two optional fields added. h3. resourcemanager_adminstration_protocol.proto * All methods for RMAdmin CLI h3. yarn_security_token.proto * Four optional fields added * Added nodeLabelExpression ** YARN-3354. Confirmed 2.8.0 and later. * Added containerType ** YARN-3116. Confirmed 2.8.0 and later. * Added executionType ** YARN-2882. Confirmed 2.9.0 and later. * Added version ** YARN-5221. Confirmed 2.8.0 and later. h3. yarn_server_common_protos.proto * Four optional fields added h3. yarn_server_nodemanager_recovery.proto * All optional fields and self-contained message changes h3. yarn_server_resourcemanager_recovery.proto * All optional fields and self-contained message changes h2. New proto files (i.e. ignorable) * yarn_server_federation_protos.proto h2. Still to complete * Security.proto * yarn_protos.proto * yarn_server_common_service_protos.proto * yarn_server_resourcemanager_service_protos.proto * yarn_service_protos.proto > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852124#comment-15852124 ] Andrew Wang commented on YARN-6142: --- Thanks Wangda for the quick analysis and follow-up! > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852080#comment-15852080 ] Wangda Tan commented on YARN-6142: -- Filed YARN-6143 > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852076#comment-15852076 ] Wangda Tan commented on YARN-6142: -- Thanks [~andrew.wang] for looking at this issue, I just look at compatibility doc and patch of YARN-3583, it's my bad. We only followed the Java API compatibility but somehow missed proto compatibility. Will file sub JIRA to fix the issue. > Support rolling upgrade between 2.x and 3.x > --- > > Key: YARN-6142 > URL: https://issues.apache.org/jira/browse/YARN-6142 > Project: Hadoop YARN > Issue Type: Task > Components: rolling upgrade >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Ray Chiang >Priority: Blocker > > Counterpart JIRA to HDFS-11096. We need to: > * examine YARN and MR's JACC report for binary and source incompatibilities > * run the [PB > differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405] > that Sean wrote for HDFS-11096 for the YARN PBs. > * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are > automated and something we can run upstream. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6142) Support rolling upgrade between 2.x and 3.x
[
https://issues.apache.org/jira/browse/YARN-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852009#comment-15852009
]
Andrew Wang commented on YARN-6142:
---
I want to highlight a possible YARN issue that [~mackrorysd] found while
working on HDFS-11096. We haven't gotten a response from a YARN person and it
affects 2.8:
{quote}
For YARN (CC Karthik Kambatla), the field nodeLabels in several structures in
yarn_protos changed from string to a custom type, and memory in ResourceProto
change from int32 to int64 (not sure if it's the case in protobuf, but that may
not be incompatible?)
{quote}
[~sunilg] / [~wangda], could you help check this as you worked on YARN-3583
where this change was made? I didn't see a compatibility discussion on that
JIRA.
> Support rolling upgrade between 2.x and 3.x
> ---
>
> Key: YARN-6142
> URL: https://issues.apache.org/jira/browse/YARN-6142
> Project: Hadoop YARN
> Issue Type: Task
> Components: rolling upgrade
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Ray Chiang
>Priority: Blocker
>
> Counterpart JIRA to HDFS-11096. We need to:
> * examine YARN and MR's JACC report for binary and source incompatibilities
> * run the [PB
> differ|https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405]
> that Sean wrote for HDFS-11096 for the YARN PBs.
> * sanity test some rolling upgrades between 2.x and 3.x. Ideally these are
> automated and something we can run upstream.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
