Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog merged PR #427: URL: https://github.com/apache/tez/pull/427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
zhangbutao commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3748958720 Thank you! @abstractdog I can't wait to experience the features of Tez on Cloud. Let's move on to the next step! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3749375975
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 37s | | master passed |
| +1 :green_heart: | compile | 5m 24s | | master passed |
| +1 :green_heart: | checkstyle | 3m 18s | | master passed |
| +1 :green_heart: | javadoc | 3m 58s | | master passed |
| -1 :x: | spotbugs | 1m 19s |
[/branch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-tez-api.txt)
| tez-api in master failed. |
| -1 :x: | spotbugs | 0m 30s |
[/branch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-tez-common.txt)
| tez-common in master failed. |
| -1 :x: | spotbugs | 0m 37s |
[/branch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in master failed. |
| -1 :x: | spotbugs | 0m 30s |
[/branch-spotbugs-tez-examples.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-tez-examples.txt)
| tez-examples in master failed. |
| -1 :x: | spotbugs | 0m 44s |
[/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-tez-dag.txt)
| tez-dag in master failed. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/branch-spotbugs-root.txt)
| root in master failed. |
| -0 :warning: | patch | 12m 41s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 22s | | the patch passed |
| +1 :green_heart: | codespell | 0m 51s | | No new issues. |
| +1 :green_heart: | compile | 5m 28s | | the patch passed |
| +1 :green_heart: | javac | 5m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 28s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 23s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 33s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 3m 58s | | the patch passed |
| -1 :x: | spotbugs | 0m 38s |
[/patch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/patch-spotbugs-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | spotbugs | 0m 31s |
[/patch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/patch-spotbugs-tez-common.txt)
| tez-common in the patch failed. |
| -1 :x: | spotbugs | 0m 38s |
[/patch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/patch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in the patch failed. |
| -1 :x: | spotbugs |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3748793917 Thanks, @ayushtkn, for the approval! I'm about to merge this soon, let me wait another few hours to maybe let @okumin, @zhangbutao chime in if I addressed all their comments regarding the code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3748284022 > hm, something happened to spotbugs, even the latest version didn't help: [0f9a678](https://github.com/apache/tez/commit/0f9a6789796d2e66ed16363e9ed4c30f205a52b5) created [TEZ-4669](https://issues.apache.org/jira/browse/TEZ-4669) about this, assuming spotbugs to be clean with this patch according to earlier runs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3738550613 hm, something happened to spotbugs, even the latest version didn't help: https://github.com/apache/tez/pull/427/commits/0f9a6789796d2e66ed16363e9ed4c30f205a52b5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3738387978
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 32s | | master passed |
| +1 :green_heart: | compile | 5m 24s | | master passed |
| +1 :green_heart: | checkstyle | 3m 17s | | master passed |
| +1 :green_heart: | javadoc | 4m 5s | | master passed |
| -1 :x: | spotbugs | 1m 15s |
[/branch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-tez-api.txt)
| tez-api in master failed. |
| -1 :x: | spotbugs | 0m 29s |
[/branch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-tez-common.txt)
| tez-common in master failed. |
| -1 :x: | spotbugs | 0m 38s |
[/branch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in master failed. |
| -1 :x: | spotbugs | 0m 31s |
[/branch-spotbugs-tez-examples.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-tez-examples.txt)
| tez-examples in master failed. |
| -1 :x: | spotbugs | 0m 43s |
[/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-tez-dag.txt)
| tez-dag in master failed. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/branch-spotbugs-root.txt)
| root in master failed. |
| -0 :warning: | patch | 8m 59s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 20s | | the patch passed |
| +1 :green_heart: | codespell | 0m 51s | | No new issues. |
| +1 :green_heart: | compile | 5m 25s | | the patch passed |
| +1 :green_heart: | javac | 5m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 27s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 24s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | hadolint | 0m 1s | | No new issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 3m 57s | | the patch passed |
| +1 :green_heart: | spotbugs | 13m 41s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 35s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 43s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 12s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 31s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 5m 5
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2681539466
##
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##
@@ -513,19 +514,21 @@ protected void serviceInit(final Configuration conf)
throws Exception {
jobTokenSecretManager = new JobTokenSecretManager(amConf);
-sessionToken =
-TokenCache.getSessionToken(amCredentials);
+sessionToken = frameworkService.getAMExtensions().getSessionToken(
+ appAttemptID, jobTokenSecretManager, amCredentials);
+
if (sessionToken == null) {
- throw new RuntimeException("Could not find session token in AM
Credentials");
+ sessionToken = TokenCache.getSessionToken(amCredentials);
Review Comment:
right, so the extra attempt to fetch it from session token doesn't make
sense, let me change this to:
```
if (sessionToken == null) {
throw new RuntimeException("Could not find session token in AM
Credentials");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2681539466
##
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##
@@ -513,19 +514,21 @@ protected void serviceInit(final Configuration conf)
throws Exception {
jobTokenSecretManager = new JobTokenSecretManager(amConf);
-sessionToken =
-TokenCache.getSessionToken(amCredentials);
+sessionToken = frameworkService.getAMExtensions().getSessionToken(
+ appAttemptID, jobTokenSecretManager, amCredentials);
+
if (sessionToken == null) {
- throw new RuntimeException("Could not find session token in AM
Credentials");
+ sessionToken = TokenCache.getSessionToken(amCredentials);
Review Comment:
right, so the extra effort doesn't make sense, let me change this to:
```
if (sessionToken == null) {
throw new RuntimeException("Could not find session token in AM
Credentials");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2681528387
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneClientFrameworkService.java:
##
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.zookeeper.ZkFrameworkClient;
+import org.apache.tez.frameworkplugins.ClientFrameworkService;
+
+public class ZkStandaloneClientFrameworkService implements
ClientFrameworkService {
+ @Override public FrameworkClient newFrameworkClient() {
Review Comment:
good catch, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3737689145 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
okumin commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2681187034
##
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
##
@@ -513,19 +514,21 @@ protected void serviceInit(final Configuration conf)
throws Exception {
jobTokenSecretManager = new JobTokenSecretManager(amConf);
-sessionToken =
-TokenCache.getSessionToken(amCredentials);
+sessionToken = frameworkService.getAMExtensions().getSessionToken(
+ appAttemptID, jobTokenSecretManager, amCredentials);
+
if (sessionToken == null) {
- throw new RuntimeException("Could not find session token in AM
Credentials");
+ sessionToken = TokenCache.getSessionToken(amCredentials);
Review Comment:
When does this double-check help?
- YARN -> When `sessionToken` is null,
`TokenCache.getSessionToken(amCredentials)` is probably always null
- ZK -> `sessionToken` is not always null
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneClientFrameworkService.java:
##
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.zookeeper.ZkFrameworkClient;
+import org.apache.tez.frameworkplugins.ClientFrameworkService;
+
+public class ZkStandaloneClientFrameworkService implements
ClientFrameworkService {
+ @Override public FrameworkClient newFrameworkClient() {
Review Comment:
```suggestion
@Override
public FrameworkClient newFrameworkClient() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3732388634
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 32s | | master passed |
| +1 :green_heart: | compile | 5m 25s | | master passed |
| +1 :green_heart: | checkstyle | 3m 16s | | master passed |
| +1 :green_heart: | javadoc | 4m 6s | | master passed |
| -1 :x: | spotbugs | 1m 17s |
[/branch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-tez-api.txt)
| tez-api in master failed. |
| -1 :x: | spotbugs | 0m 31s |
[/branch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-tez-common.txt)
| tez-common in master failed. |
| -1 :x: | spotbugs | 0m 37s |
[/branch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in master failed. |
| -1 :x: | spotbugs | 0m 31s |
[/branch-spotbugs-tez-examples.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-tez-examples.txt)
| tez-examples in master failed. |
| -1 :x: | spotbugs | 0m 42s |
[/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-tez-dag.txt)
| tez-dag in master failed. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/branch-spotbugs-root.txt)
| root in master failed. |
| -0 :warning: | patch | 12m 45s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 24s | | the patch passed |
| +1 :green_heart: | codespell | 0m 51s | | No new issues. |
| +1 :green_heart: | compile | 5m 26s | | the patch passed |
| +1 :green_heart: | javac | 5m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 24s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 29s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 22s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 45s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/39/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed |
| +1 :green_heart: | spotbugs | 13m 39s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 37s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 43s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 10s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 32s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 6m 21s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 73m 1s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 47s | | The patch does not
generate ASF License warni
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3729121113
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 31s | | master passed |
| +1 :green_heart: | compile | 5m 22s | | master passed |
| +1 :green_heart: | checkstyle | 3m 16s | | master passed |
| +1 :green_heart: | javadoc | 4m 5s | | master passed |
| -1 :x: | spotbugs | 1m 18s |
[/branch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-tez-api.txt)
| tez-api in master failed. |
| -1 :x: | spotbugs | 0m 30s |
[/branch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-tez-common.txt)
| tez-common in master failed. |
| -1 :x: | spotbugs | 0m 38s |
[/branch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in master failed. |
| -1 :x: | spotbugs | 0m 31s |
[/branch-spotbugs-tez-examples.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-tez-examples.txt)
| tez-examples in master failed. |
| -1 :x: | spotbugs | 0m 44s |
[/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-tez-dag.txt)
| tez-dag in master failed. |
| -1 :x: | spotbugs | 0m 23s |
[/branch-spotbugs-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/branch-spotbugs-root.txt)
| root in master failed. |
| -0 :warning: | patch | 12m 46s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 24s | | the patch passed |
| +1 :green_heart: | codespell | 0m 50s | | No new issues. |
| +1 :green_heart: | compile | 5m 28s | | the patch passed |
| +1 :green_heart: | javac | 5m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 28s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 23s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 3m 58s | | the patch passed |
| -1 :x: | spotbugs | 0m 37s |
[/patch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/patch-spotbugs-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | spotbugs | 0m 30s |
[/patch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/patch-spotbugs-tez-common.txt)
| tez-common in the patch failed. |
| -1 :x: | spotbugs | 0m 38s |
[/patch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/38/artifact/out/patch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in the patch failed. |
| -1 :x: | spotbugs |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3728629713 @ayushtkn : address your comments, good catches, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2675939284
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ // Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+
+ private CuratorFramework client;
+ private TreeCache cache;
+ private ZkRegistryListener listener;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// Not a leaf path. Only leaf path contains AMRecord.
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, StandardCharsets.UTF_8);
+try {
+ return AMRegistryUtils.jsonStringToRecord(value);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFramework();
+cache = new TreeCache(client, zkConf.getZkNamespace());
+client.start();
+cache.start();
+listen
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3726034661
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 23m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 1s | | master passed |
| +1 :green_heart: | compile | 5m 28s | | master passed |
| +1 :green_heart: | checkstyle | 3m 18s | | master passed |
| +1 :green_heart: | javadoc | 3m 59s | | master passed |
| -1 :x: | spotbugs | 1m 18s |
[/branch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-tez-api.txt)
| tez-api in master failed. |
| -1 :x: | spotbugs | 0m 30s |
[/branch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-tez-common.txt)
| tez-common in master failed. |
| -1 :x: | spotbugs | 0m 38s |
[/branch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in master failed. |
| -1 :x: | spotbugs | 0m 30s |
[/branch-spotbugs-tez-examples.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-tez-examples.txt)
| tez-examples in master failed. |
| -1 :x: | spotbugs | 0m 43s |
[/branch-spotbugs-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-tez-dag.txt)
| tez-dag in master failed. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/branch-spotbugs-root.txt)
| root in master failed. |
| -0 :warning: | patch | 12m 42s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 22s | | the patch passed |
| +1 :green_heart: | codespell | 0m 50s | | No new issues. |
| +1 :green_heart: | compile | 5m 26s | | the patch passed |
| +1 :green_heart: | javac | 5m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 28s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 23s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 45s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 3m 58s | | the patch passed |
| -1 :x: | spotbugs | 0m 38s |
[/patch-spotbugs-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/patch-spotbugs-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | spotbugs | 0m 30s |
[/patch-spotbugs-tez-common.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/patch-spotbugs-tez-common.txt)
| tez-common in the patch failed. |
| -1 :x: | spotbugs | 0m 34s |
[/patch-spotbugs-tez-runtime-library.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/37/artifact/out/patch-spotbugs-tez-runtime-library.txt)
| tez-runtime-library in the patch failed. |
| -1 :x: | spotbugs |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2672453303
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private static final String SERVER_FRAMEWORK_SERVICE_INTERFACE_NAME =
+ "org.apache.tez.frameworkplugins.ServerFrameworkService";
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise: the default class will be instantiated and returned
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf,
+ Class defaultClazz) {
+String modeInConf = conf != null ?
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) : null;
+String modeInEnv = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+try {
+ if (modeInConf != null) {
Review Comment:
ack, much cleaner and takes care of empty string too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2672423322
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.google.common.annotations.VisibleForTesting;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
Review Comment:
it should, also, tracking `amHost` and `amPort` separately doesn't make
sense, let's just rely on a single volatile record field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2672393297
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ // Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+
+ private CuratorFramework client;
+ private TreeCache cache;
+ private ZkRegistryListener listener;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// Not a leaf path. Only leaf path contains AMRecord.
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, StandardCharsets.UTF_8);
+try {
+ return AMRegistryUtils.jsonStringToRecord(value);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFramework();
+cache = new TreeCache(client, zkConf.getZkNamespace());
+client.start();
+cache.start();
Review Com
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2672066145
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ // Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+
+ private CuratorFramework client;
+ private TreeCache cache;
+ private ZkRegistryListener listener;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// Not a leaf path. Only leaf path contains AMRecord.
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, StandardCharsets.UTF_8);
+try {
+ return AMRegistryUtils.jsonStringToRecord(value);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFramework();
+cache = new TreeCache(client, zkConf.getZkNamespace());
+client.start();
+cache.start();
+listen
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2672066145
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ // Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+
+ private CuratorFramework client;
+ private TreeCache cache;
+ private ZkRegistryListener listener;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// Not a leaf path. Only leaf path contains AMRecord.
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, StandardCharsets.UTF_8);
+try {
+ return AMRegistryUtils.jsonStringToRecord(value);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFramework();
+cache = new TreeCache(client, zkConf.getZkNamespace());
+client.start();
+cache.start();
+listen
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671865869
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ // Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+
+ private CuratorFramework client;
+ private TreeCache cache;
+ private ZkRegistryListener listener;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// Not a leaf path. Only leaf path contains AMRecord.
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, StandardCharsets.UTF_8);
+try {
+ return AMRegistryUtils.jsonStringToRecord(value);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFramework();
+cache = new TreeCache(client, zkConf.getZkNamespace());
+client.start();
+cache.start();
+listen
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671856385
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(AMRegistryClient.class);
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Lookup AM metadata for the given application ID.
+ *
+ * @param appId the application ID
+ * @return the AM record if found, otherwise {@code null}
+ * @throws IOException if the lookup fails
+ */
+ public abstract AMRecord getRecord(ApplicationId appId) throws IOException;
+
+ /**
+ * Retrieve all AM records known in the registry.
+ *
+ * @return a list of AM records (possibly empty)
+ * @throws IOException if the fetch fails
+ */
+ public abstract List getAllRecords() throws IOException;
+
+ /**
+ * Register a listener for AM registry events.
+ * The listener will be notified when AM records are added or removed.
+ *
+ * @param listener the listener to add
+ */
+ public synchronized void addListener(AMRegistryClientListener listener) {
+listeners.add(listener);
+ }
+
+ /**
+ * Notify listeners of a newly added AM record.
+ *
+ * @param record the added AM record
+ */
+ protected synchronized void notifyOnAdded(AMRecord record) {
+for (AMRegistryClientListener listener : listeners) {
+ try {
+listener.onAdd(record);
+ } catch (Exception e) {
+LOG.warn("Exception while calling AM add listener, AM record {}",
record, e);
+ }
+}
+ }
+
+ /**
+ * Notify listeners of an updated AM record.
+ *
+ * @param record the updated AM record
+ */
+ protected synchronized void notifyOnUpdated(AMRecord record) {
+for (AMRegistryClientListener listener : listeners) {
+ try {
+listener.onUpdate(record);
+ } catch (Exception e) {
+LOG.warn("Exception while calling AM update listener, AM record {}",
record, e);
Review Comment:
yeah, and similar change can be done for the other messages in this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671851521
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(AMRegistryClient.class);
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Lookup AM metadata for the given application ID.
+ *
+ * @param appId the application ID
+ * @return the AM record if found, otherwise {@code null}
+ * @throws IOException if the lookup fails
+ */
+ public abstract AMRecord getRecord(ApplicationId appId) throws IOException;
+
+ /**
+ * Retrieve all AM records known in the registry.
+ *
+ * @return a list of AM records (possibly empty)
+ * @throws IOException if the fetch fails
+ */
+ public abstract List getAllRecords() throws IOException;
+
+ /**
+ * Register a listener for AM registry events.
+ * The listener will be notified when AM records are added or removed.
+ *
+ * @param listener the listener to add
+ */
+ public synchronized void addListener(AMRegistryClientListener listener) {
+listeners.add(listener);
+ }
+
Review Comment:
simply hasn't needed such a thing so far, currently, we just subscribe and
close the client in the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671834345
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
Review Comment:
we can, but it doesn't make the code better: this class is usually used as
field, not local variable in try-catch, this applies to production as well as
testing code, so I think Closeable might be better here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671733389
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java:
##
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+
+/**
+ * Base class for {@code AMRegistry} implementations.
+ *
+ * The specific implementation is configured via the
+ * {@code tez.am.registry.class} property.
+ *
+ * Implementations are expected to provide appropriate service lifecycle
+ * behavior, including:
+ *
+ * {@code init}
+ * {@code serviceStart}
+ * {@code serviceStop}
+ *
+ *
+ */
+public interface AMRegistry extends AutoCloseable {
+
+ void add(AMRecord server) throws Exception;
+
+ void remove(AMRecord server) throws Exception;
Review Comment:
record is better, fixing this
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java:
##
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+
+/**
+ * Base class for {@code AMRegistry} implementations.
+ *
+ * The specific implementation is configured via the
+ * {@code tez.am.registry.class} property.
+ *
+ * Implementations are expected to provide appropriate service lifecycle
+ * behavior, including:
+ *
+ * {@code init}
+ * {@code serviceStart}
+ * {@code serviceStop}
+ *
+ *
+ */
+public interface AMRegistry extends AutoCloseable {
+
+ void add(AMRecord server) throws Exception;
+
+ void remove(AMRecord server) throws Exception;
+
+ ApplicationId generateNewId() throws Exception;
+
+ AMRecord createAmRecord(ApplicationId appId, String hostName, String hostIp,
int port,
+ String computeName);
+
+ void close();
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2671729361
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -125,9 +154,11 @@ public boolean equals(Object other) {
}
if (other instanceof AMRecord otherRecord) {
return appId.equals(otherRecord.appId)
- && host.equals(otherRecord.host)
+ && hostName.equals(otherRecord.hostName)
Review Comment:
this is initialized from DAGAppMaster:
https://github.com/apache/tez/pull/427/files#diff-54ba4a2af15261379079ed5a9c1f9eea52da7bdd3f1109fe7b96d4abf07f6173R677-R678
```
AMRecord amRecord = amRegistry.createAmRecord(appId,
rpcServerAddress.getHostName(),
rpcServerAddress.getAddress().getHostAddress(),
rpcServerAddress.getPort(), computeName);
```
hostName is not supposed to be null, so if it's null, it's already a fatal
issue, which is revealed by NPE, I don't think we make this better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2669024497
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -148,16 +179,27 @@ public boolean equals(Object other) {
* @return a {@link ServiceRecord} populated with the values of this {@code
AMRecord}
*/
public ServiceRecord toServiceRecord() {
-ServiceRecord serviceRecord = new ServiceRecord();
+if (serviceRecord != null) {
+ return serviceRecord;
+}
+serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
-serviceRecord.set(HOST_RECORD_KEY, host);
+serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
+serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
-serviceRecord.set(OPAQUE_ID_KEY, id);
+serviceRecord.set(EXTERNAL_ID_KEY, externalId);
+serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
+
return serviceRecord;
}
+ @Override
+ public String toString() {
+return toServiceRecord().attributes().toString();
Review Comment:
I think there is no "before setting the values" scenario given only final
fields, I'm assuming the the logging must happen with the AMRecord instance
when the constructor is definitely finished
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2669024497
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -148,16 +179,27 @@ public boolean equals(Object other) {
* @return a {@link ServiceRecord} populated with the values of this {@code
AMRecord}
*/
public ServiceRecord toServiceRecord() {
-ServiceRecord serviceRecord = new ServiceRecord();
+if (serviceRecord != null) {
+ return serviceRecord;
+}
+serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
-serviceRecord.set(HOST_RECORD_KEY, host);
+serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
+serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
-serviceRecord.set(OPAQUE_ID_KEY, id);
+serviceRecord.set(EXTERNAL_ID_KEY, externalId);
+serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
+
return serviceRecord;
}
+ @Override
+ public String toString() {
+return toServiceRecord().attributes().toString();
Review Comment:
I think there is no "before setting the values" scenario given only final
fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
ayushtkn commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2609634048
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
Review Comment:
can we do ``implements AutoCloseable``?
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -125,9 +154,11 @@ public boolean equals(Object other) {
}
if (other instanceof AMRecord otherRecord) {
return appId.equals(otherRecord.appId)
- && host.equals(otherRecord.host)
+ && hostName.equals(otherRecord.hostName)
Review Comment:
can this lead to NPE, we don't have any default here
```
this.hostName = serviceRecord.get(HOST_NAME_RECORD_KEY);
```
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.google.common.annotations.VisibleForTesting;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
+ private ZkAMRegistryClient amRegistryClient = null;
+ private volatile boolean isRunning = false;
+ private String amHost;
+ private int amPort;
+
+ @Override
+ public synchronized void init(TezConfiguration tezConf) {
+if (amRegistryClient == null) {
+ try {
+amRegistryClient = ZkAMRegistryClient.getClient(tezConf);
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ amRegistryClient.start();
Review Comment:
we should return if `isRunning` is ` true` here
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -148,16 +179,27 @@ public boolean equals(Object other) {
* @ret
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3601689298
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 51s | | master passed |
| +1 :green_heart: | compile | 5m 23s | | master passed |
| +1 :green_heart: | checkstyle | 3m 14s | | master passed |
| +1 :green_heart: | javadoc | 4m 3s | | master passed |
| +0 :ok: | spotbugs | 2m 9s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 47s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 10s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 45s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 44s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 6m 51s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 34s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 21s | | the patch passed |
| +1 :green_heart: | codespell | 0m 50s | | No new issues. |
| +1 :green_heart: | compile | 5m 19s | | the patch passed |
| +1 :green_heart: | javac | 5m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 27s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 23s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 33s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 45s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 13s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 36s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 42s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 9s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 32s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 6m 7s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 73m 45s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 46s | | The patch does not
generate ASF License warnings. |
| | | 170m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux e3f0c21725bf 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3598374712
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 59s | | master passed |
| +1 :green_heart: | compile | 5m 15s | | master passed |
| +1 :green_heart: | checkstyle | 2m 54s | | master passed |
| +1 :green_heart: | javadoc | 4m 26s | | master passed |
| +0 :ok: | spotbugs | 2m 11s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 47s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 11s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 41s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 45s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 21s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 8m 6s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 53s | | the patch passed |
| +1 :green_heart: | codespell | 0m 51s | | No new issues. |
| +1 :green_heart: | compile | 5m 20s | | the patch passed |
| +1 :green_heart: | javac | 5m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 22s |
[/results-checkstyle-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/34/artifact/out/results-checkstyle-tez-api.txt)
| tez-api: The patch generated 1 new + 118 unchanged - 3 fixed = 119 total
(was 121) |
| +1 :green_heart: | checkstyle | 0m 21s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 21s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 19s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/34/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/34/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 740 unchanged - 6 fixed = 743 total (was
746) |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 43s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 35s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 40s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 5m 58s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 26s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 6m 1s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 73m 2s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 32s | | The patch does not
generate ASF License warnings. |
| | | 170m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/34/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux dee85b6ffdbd 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3598106074
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 8m 1s | | master passed |
| +1 :green_heart: | compile | 2m 50s | | master passed |
| +1 :green_heart: | checkstyle | 1m 34s | | master passed |
| +1 :green_heart: | javadoc | 2m 3s | | master passed |
| +0 :ok: | spotbugs | 1m 29s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 24s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 38s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 23s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 51s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 3m 47s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 4m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 54s | | the patch passed |
| +1 :green_heart: | codespell | 0m 31s | | No new issues. |
| +1 :green_heart: | compile | 2m 49s | | the patch passed |
| +1 :green_heart: | javac | 2m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | tez-api: The patch
generated 0 new + 117 unchanged - 3 fixed = 117 total (was 120) |
| +1 :green_heart: | checkstyle | 0m 11s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 13s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 11s | | tez-examples: The patch
generated 0 new + 11 unchanged - 1 fixed = 11 total (was 12) |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/35/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total
(was 612) |
| -0 :warning: | checkstyle | 0m 22s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/35/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed |
| +1 :green_heart: | spotbugs | 7m 39s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 1s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 25s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 4m 6s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 20s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 4m 33s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 60m 3s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 28s | | The patch does not
generate ASF License warnings. |
| | | 119m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/35/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux b839cd43c9e4 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9
00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3596352596
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 7m 47s | | master passed |
| +1 :green_heart: | compile | 2m 54s | | master passed |
| +1 :green_heart: | checkstyle | 1m 45s | | master passed |
| +1 :green_heart: | javadoc | 2m 11s | | master passed |
| +0 :ok: | spotbugs | 1m 28s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 27s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 38s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 26s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 53s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 3m 40s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 4m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 41s | | the patch passed |
| +1 :green_heart: | codespell | 0m 29s | | No new issues. |
| +1 :green_heart: | compile | 2m 56s | | the patch passed |
| +1 :green_heart: | javac | 2m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | tez-api: The patch
generated 0 new + 117 unchanged - 3 fixed = 117 total (was 120) |
| +1 :green_heart: | checkstyle | 0m 12s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 14s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 12s | | tez-examples: The patch
generated 0 new + 11 unchanged - 1 fixed = 11 total (was 12) |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/33/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/33/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed |
| +1 :green_heart: | spotbugs | 7m 37s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 2s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 27s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 4m 16s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 18s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 4m 52s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 59m 20s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 43s | | The patch does not
generate ASF License warnings. |
| | | 133m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/33/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux d19566a5022f 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9
00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-3595442285 @ayushtkn : all your comments have been addressed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3594118860
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 0s | | master passed |
| +1 :green_heart: | compile | 5m 22s | | master passed |
| +1 :green_heart: | checkstyle | 3m 18s | | master passed |
| +1 :green_heart: | javadoc | 4m 3s | | master passed |
| +0 :ok: | spotbugs | 2m 8s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 46s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 14s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 44s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 46s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 6m 58s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 43s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 32s | | the patch passed |
| +1 :green_heart: | codespell | 0m 50s | | No new issues. |
| +1 :green_heart: | compile | 5m 25s | | the patch passed |
| +1 :green_heart: | javac | 5m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 28s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 22s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/31/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/31/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 57s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 14s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 36s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 44s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 9s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 31s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 6m 2s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 72m 14s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 42s | | The patch does not
generate ASF License warnings. |
| | | 169m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/31/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux 7c7c4bb63872 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3591711761
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 9m 52s | | master passed |
| +1 :green_heart: | compile | 5m 24s | | master passed |
| +1 :green_heart: | checkstyle | 3m 16s | | master passed |
| +1 :green_heart: | javadoc | 4m 4s | | master passed |
| +0 :ok: | spotbugs | 2m 10s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 45s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 11s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 43s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 44s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 6m 56s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 40s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 19s | | the patch passed |
| +1 :green_heart: | codespell | 0m 50s | | No new issues. |
| +1 :green_heart: | compile | 5m 17s | | the patch passed |
| +1 :green_heart: | javac | 5m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 21s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 26s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 21s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 33s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/30/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 44s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/30/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 55s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 10s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 37s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 44s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 8s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 32s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 6m 28s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 74m 14s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 46s | | The patch does not
generate ASF License warnings. |
| | | 171m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/30/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux de6817b7aa0a 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3585935418
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 13s | | master passed |
| +1 :green_heart: | compile | 5m 10s | | master passed |
| +1 :green_heart: | checkstyle | 3m 3s | | master passed |
| +1 :green_heart: | javadoc | 3m 55s | | master passed |
| +0 :ok: | spotbugs | 2m 12s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 45s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 11s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 43s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 49s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 31s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 8m 17s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 46s | | the patch passed |
| +1 :green_heart: | codespell | 0m 53s | | No new issues. |
| +1 :green_heart: | compile | 5m 32s | | the patch passed |
| +1 :green_heart: | javac | 5m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 20s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 26s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 20s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed |
| -1 :x: | spotbugs | 1m 43s |
[/new-spotbugs-tez-api.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/new-spotbugs-tez-api.html)
| tez-api generated 5 new + 610 unchanged - 0 fixed = 615 total (was 610) |
| -1 :x: | spotbugs | 7m 45s |
[/new-spotbugs-root.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/new-spotbugs-root.html)
| root generated 5 new + 2066 unchanged - 0 fixed = 2071 total (was 2066) |
_ Other Tests _ |
| -1 :x: | unit | 2m 24s |
[/patch-unit-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/patch-unit-tez-api.txt)
| tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 40s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 4s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 31s | | tez-examples in the patch
passed. |
| -1 :x: | unit | 6m 2s |
[/patch-unit-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/patch-unit-tez-dag.txt)
| tez-dag in the patch passed. |
| -1 :x: | unit | 74m 41s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/29/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3585542172
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 8s | | master passed |
| +1 :green_heart: | compile | 5m 23s | | master passed |
| +1 :green_heart: | checkstyle | 2m 57s | | master passed |
| +1 :green_heart: | javadoc | 3m 47s | | master passed |
| +0 :ok: | spotbugs | 2m 10s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 46s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 11s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 41s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 46s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 29s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 8m 10s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 55s | | the patch passed |
| +1 :green_heart: | codespell | 1m 0s | | No new issues. |
| +1 :green_heart: | compile | 5m 18s | | the patch passed |
| +1 :green_heart: | javac | 5m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/results-checkstyle-tez-api.txt)
| tez-api: The patch generated 1 new + 118 unchanged - 3 fixed = 119 total
(was 121) |
| +1 :green_heart: | checkstyle | 0m 20s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 18s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 32s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was
746) |
| -1 :x: | javadoc | 0m 39s |
[/results-javadoc-javadoc-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/results-javadoc-javadoc-tez-api.txt)
| tez-api generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) |
| -1 :x: | javadoc | 1m 21s |
[/results-javadoc-javadoc-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/results-javadoc-javadoc-root.txt)
| root generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) |
| -1 :x: | spotbugs | 1m 46s |
[/new-spotbugs-tez-api.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/new-spotbugs-tez-api.html)
| tez-api generated 5 new + 610 unchanged - 0 fixed = 615 total (was 610) |
| -1 :x: | spotbugs | 7m 31s |
[/new-spotbugs-root.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/new-spotbugs-root.html)
| root generated 5 new + 2066 unchanged - 0 fixed = 2071 total (was 2066) |
_ Other Tests _ |
| -1 :x: | unit | 2m 25s |
[/patch-unit-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/28/artifact/out/patch-unit-tez-api.txt)
| tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 43s | | tez-common in the patch passed.
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3585167290
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 3s | | master passed |
| +1 :green_heart: | compile | 5m 19s | | master passed |
| +1 :green_heart: | checkstyle | 3m 1s | | master passed |
| +1 :green_heart: | javadoc | 3m 42s | | master passed |
| +0 :ok: | spotbugs | 2m 11s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 43s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 10s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 40s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 49s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 34s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 8m 16s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-mvninstall-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | mvninstall | 0m 26s |
[/patch-mvninstall-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-mvninstall-tez-dag.txt)
| tez-dag in the patch failed. |
| -1 :x: | mvninstall | 1m 42s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | codespell | 0m 55s | | No new issues. |
| -1 :x: | compile | 0m 36s |
[/patch-compile-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | compile | 0m 37s |
[/patch-compile-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-tez-dag.txt)
| tez-dag in the patch failed. |
| -1 :x: | compile | 0m 37s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| -1 :x: | javac | 0m 36s |
[/patch-compile-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-tez-api.txt)
| tez-api in the patch failed. |
| -1 :x: | javac | 0m 37s |
[/patch-compile-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-tez-dag.txt)
| tez-dag in the patch failed. |
| -1 :x: | javac | 0m 37s |
[/patch-compile-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/patch-compile-root.txt)
| root in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 22s |
[/results-checkstyle-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/results-checkstyle-tez-api.txt)
| tez-api: The patch generated 1 new + 118 unchanged - 3 fixed = 119 total
(was 121) |
| +1 :green_heart: | checkstyle | 0m 18s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 24s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 19s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/27/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unc
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2565261793 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) + .forPath(namespace + "/" + server.getApplicationId().toString(), json.getBytes(StandardCharsets.UTF_8)); +} +amRecords.add(server); + } + + @Override + public void remove(AMRecord server) throws Exception { +amRecords.remove(server); +final String path = namespace + "/" + server.getApplicationId().toString(); +client.delete().forPath(path); +LOG.info("Deleted AMRecord from zkpath {}", path); + } + + @Override + public ApplicationId generateNewId() throws Exception { +createNamespaceIfNotExists(); +long namespaceCreationTime = getNamespaceCreationTime(); + +boolean success = false; +long startTime = System.currentTimeMillis(); +RetryPolicy retryPolicy = zkConfig.getRetryPolicy(); +int tryId = 0; +for (int i = 0; (i < zkConfig.getCuratorMaxRetries()) &
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2567667239 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); Review Comment: yeah, fixed as: ``` private final List amRecords = Collections.synchronizedList(new ArrayList<>()); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2567618554
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneServerFrameworkService.java:
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistry;
+import org.apache.tez.frameworkplugins.AMExtensions;
+import org.apache.tez.frameworkplugins.ServerFrameworkService;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ZkStandaloneServerFrameworkService implements
ServerFrameworkService {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkStandaloneServerFrameworkService.class);
+ private ZkAMRegistry amRegistry;
+
+ @Override
+ public synchronized AMRegistry getAMRegistry(Configuration conf) {
+if (amRegistry == null) {
+ try {
+final String externalID =
System.getenv(TezConstants.TEZ_AM_EXTERNAL_ID);
+amRegistry = new ZkAMRegistry(externalID);
+amRegistry.init(conf);
+amRegistry.start();
Review Comment:
added javadoc comment to clarify this:
```
/**
* Returns a singleton {@link AMRegistry} instance backed by ZooKeeper.
*
* If the registry has not yet been created, this method initializes
and starts
* a new {@link ZkAMRegistry} using the external AM identifier obtained
from the
* {@code TEZ_AM_EXTERNAL_ID} environment variable.
*
* When the registry is used as a service within the DAGAppMaster, the
* DAGAppMaster is responsible for managing its lifecycle, including
closure.
*
* @param conf the configuration used to initialize the registry; must not
be null
* @return the initialized and started {@link AMRegistry} instance
* @throws IllegalStateException if the {@code TEZ_AM_EXTERNAL_ID}
environment variable is not set
* @throws RuntimeException if an error occurs while creating,
initializing, or starting the registry
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2567607890
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneServerFrameworkService.java:
##
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistry;
+import org.apache.tez.frameworkplugins.AMExtensions;
+import org.apache.tez.frameworkplugins.ServerFrameworkService;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ZkStandaloneServerFrameworkService implements
ServerFrameworkService {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkStandaloneServerFrameworkService.class);
+ private ZkAMRegistry amRegistry;
+
+ @Override
+ public synchronized AMRegistry getAMRegistry(Configuration conf) {
+if (amRegistry == null) {
+ try {
+final String externalID =
System.getenv(TezConstants.TEZ_AM_EXTERNAL_ID);
Review Comment:
right, let me give some thoughts for the user as below:
```
final String externalId =
System.getenv(TezConstants.TEZ_AM_EXTERNAL_ID);
if (externalId == null) {
throw new IllegalStateException(
TezConstants.TEZ_AM_EXTERNAL_ID + " environment variable is not
set for standalone AM");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2567589882
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneAMExtensions.java:
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.common.TezUtilsInternal;
+import org.apache.tez.common.security.JobTokenIdentifier;
+import org.apache.tez.common.security.JobTokenSecretManager;
+import org.apache.tez.common.security.TokenCache;
+import org.apache.tez.dag.api.records.DAGProtos;
+import org.apache.tez.dag.app.ClusterInfo;
+import org.apache.tez.dag.app.dag.Vertex;
+import org.apache.tez.frameworkplugins.AMExtensions;
+import org.apache.tez.frameworkplugins.ServerFrameworkService;
+
+public class ZkStandaloneAMExtensions implements AMExtensions {
+
+ private final ServerFrameworkService frameworkService;
+
+ public ZkStandaloneAMExtensions(ServerFrameworkService frameworkService) {
+this.frameworkService = frameworkService;
+ }
+
+ @Override
+ public ContainerId allocateContainerId(Configuration conf) {
+try {
+ AMRegistry amRegistry = frameworkService.getAMRegistry(conf);
+ if (amRegistry != null) {
+ApplicationId appId = amRegistry.generateNewId();
+// attemptId is set to 1 only then APP_LAUNCHED event gets triggered
+ApplicationAttemptId applicationAttemptId =
ApplicationAttemptId.newInstance(appId, 1);
+return ContainerId.newContainerId(applicationAttemptId, 0);
+ } else {
+throw new RuntimeException("AMRegistry is required for
ZkStandaloneAmExtensions");
+ }
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ @Override
+ public void checkTaskResources(Map vertices, ClusterInfo
clusterInfo) {
+// no-op: Zookeeper-based framework current enforce task resources at the
moment
+ }
+
+ @Override
+ public DAGProtos.ConfigurationProto loadConfigurationProto() throws
IOException {
+return TezUtilsInternal.loadConfProtoFromText();
+ }
+
+ @Override
+ public Token getSessionToken(ApplicationAttemptId
appAttemptID,
+ JobTokenSecretManager jobTokenSecretManager, Credentials amCredentials) {
+JobTokenIdentifier identifier = new JobTokenIdentifier(new
Text(appAttemptID.getApplicationId().toString()));
Review Comment:
the standalone mode doesn't account for different application attempt ids
while generating the app id, we rely on the zookeeper logic implemented in
ZkAmRegistry, then deriving an artificial container ID and app attempt ID in
this class (ZkStandaloneAMExtensions.allocateContainerId), see constant app
attempt id:
```
ApplicationAttemptId applicationAttemptId =
ApplicationAttemptId.newInstance(appId, 1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2567589882
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneAMExtensions.java:
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.zookeeper;
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.common.TezUtilsInternal;
+import org.apache.tez.common.security.JobTokenIdentifier;
+import org.apache.tez.common.security.JobTokenSecretManager;
+import org.apache.tez.common.security.TokenCache;
+import org.apache.tez.dag.api.records.DAGProtos;
+import org.apache.tez.dag.app.ClusterInfo;
+import org.apache.tez.dag.app.dag.Vertex;
+import org.apache.tez.frameworkplugins.AMExtensions;
+import org.apache.tez.frameworkplugins.ServerFrameworkService;
+
+public class ZkStandaloneAMExtensions implements AMExtensions {
+
+ private final ServerFrameworkService frameworkService;
+
+ public ZkStandaloneAMExtensions(ServerFrameworkService frameworkService) {
+this.frameworkService = frameworkService;
+ }
+
+ @Override
+ public ContainerId allocateContainerId(Configuration conf) {
+try {
+ AMRegistry amRegistry = frameworkService.getAMRegistry(conf);
+ if (amRegistry != null) {
+ApplicationId appId = amRegistry.generateNewId();
+// attemptId is set to 1 only then APP_LAUNCHED event gets triggered
+ApplicationAttemptId applicationAttemptId =
ApplicationAttemptId.newInstance(appId, 1);
+return ContainerId.newContainerId(applicationAttemptId, 0);
+ } else {
+throw new RuntimeException("AMRegistry is required for
ZkStandaloneAmExtensions");
+ }
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ @Override
+ public void checkTaskResources(Map vertices, ClusterInfo
clusterInfo) {
+// no-op: Zookeeper-based framework current enforce task resources at the
moment
+ }
+
+ @Override
+ public DAGProtos.ConfigurationProto loadConfigurationProto() throws
IOException {
+return TezUtilsInternal.loadConfProtoFromText();
+ }
+
+ @Override
+ public Token getSessionToken(ApplicationAttemptId
appAttemptID,
+ JobTokenSecretManager jobTokenSecretManager, Credentials amCredentials) {
+JobTokenIdentifier identifier = new JobTokenIdentifier(new
Text(appAttemptID.getApplicationId().toString()));
Review Comment:
the standalone mode doesn't account for different application attempt ids
while generating the app id, we rely on the zookeeper logic implemented in
ZkAmRegistry, then deriving an artificial container id in this class
(ZkStandaloneAMExtensions.allocateContainerId), see constant app attempt id:
```
ApplicationAttemptId applicationAttemptId =
ApplicationAttemptId.newInstance(appId, 1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2567539463 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { Review Comment: the current logic cannot account for and handle other zookeeper-side problems, because it would make this implementation complicated, instead, a fatal problem during the AM id generation process leads to AM start failure, which is eventually retried by the environment: we're assuming that in a kubernetes env, AM start failure is retried until the desired replicas -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2567530586 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) + .forPath(namespace + "/" + server.getApplicationId().toString(), json.getBytes(StandardCharsets.UTF_8)); +} +amRecords.add(server); + } + + @Override + public void remove(AMRecord server) throws Exception { +amRecords.remove(server); +final String path = namespace + "/" + server.getApplicationId().toString(); +client.delete().forPath(path); +LOG.info("Deleted AMRecord from zkpath {}", path); + } + + @Override + public ApplicationId generateNewId() throws Exception { +createNamespaceIfNotExists(); +long namespaceCreationTime = getNamespaceCreationTime(); + +boolean success = false; +long startTime = System.currentTimeMillis(); +RetryPolicy retryPolicy = zkConfig.getRetryPolicy(); +int tryId = 0; +for (int i = 0; (i < zkConfig.getCuratorMaxRetries()) &
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2565261793 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) + .forPath(namespace + "/" + server.getApplicationId().toString(), json.getBytes(StandardCharsets.UTF_8)); +} +amRecords.add(server); + } + + @Override + public void remove(AMRecord server) throws Exception { +amRecords.remove(server); +final String path = namespace + "/" + server.getApplicationId().toString(); +client.delete().forPath(path); +LOG.info("Deleted AMRecord from zkpath {}", path); + } + + @Override + public ApplicationId generateNewId() throws Exception { +createNamespaceIfNotExists(); +long namespaceCreationTime = getNamespaceCreationTime(); + +boolean success = false; +long startTime = System.currentTimeMillis(); +RetryPolicy retryPolicy = zkConfig.getRetryPolicy(); +int tryId = 0; +for (int i = 0; (i < zkConfig.getCuratorMaxRetries()) &
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2565261793 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) + .forPath(namespace + "/" + server.getApplicationId().toString(), json.getBytes(StandardCharsets.UTF_8)); +} +amRecords.add(server); + } + + @Override + public void remove(AMRecord server) throws Exception { +amRecords.remove(server); +final String path = namespace + "/" + server.getApplicationId().toString(); +client.delete().forPath(path); +LOG.info("Deleted AMRecord from zkpath {}", path); + } + + @Override + public ApplicationId generateNewId() throws Exception { +createNamespaceIfNotExists(); +long namespaceCreationTime = getNamespaceCreationTime(); + +boolean success = false; +long startTime = System.currentTimeMillis(); +RetryPolicy retryPolicy = zkConfig.getRetryPolicy(); +int tryId = 0; +for (int i = 0; (i < zkConfig.getCuratorMaxRetries()) &
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2565261793 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) + .forPath(namespace + "/" + server.getApplicationId().toString(), json.getBytes(StandardCharsets.UTF_8)); +} +amRecords.add(server); + } + + @Override + public void remove(AMRecord server) throws Exception { +amRecords.remove(server); +final String path = namespace + "/" + server.getApplicationId().toString(); +client.delete().forPath(path); +LOG.info("Deleted AMRecord from zkpath {}", path); + } + + @Override + public ApplicationId generateNewId() throws Exception { +createNamespaceIfNotExists(); +long namespaceCreationTime = getNamespaceCreationTime(); + +boolean success = false; +long startTime = System.currentTimeMillis(); +RetryPolicy retryPolicy = zkConfig.getRetryPolicy(); +int tryId = 0; +for (int i = 0; (i < zkConfig.getCuratorMaxRetries()) &
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427: URL: https://github.com/apache/tez/pull/427#discussion_r2565205502 ## tez-dag/src/main/java/org/apache/tez/dag/api/client/registry/zookeeper/ZkAMRegistry.java: ## @@ -0,0 +1,175 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tez.dag.api.client.registry.zookeeper; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.apache.curator.RetryLoop; +import org.apache.curator.RetryPolicy; +import org.apache.curator.framework.CuratorFramework; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.registry.client.binding.RegistryUtils; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.tez.client.registry.AMRecord; +import org.apache.tez.client.registry.AMRegistry; +import org.apache.tez.client.registry.zookeeper.ZkConfig; +import org.apache.zookeeper.CreateMode; +import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.data.Stat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Curator/Zookeeper impl of AMRegistry (for internal use only) + * Clients should use org.apache.tez.dag.api.client.registry.zookeeper.ZkAMRegistryClient instead. + */ [email protected] +public class ZkAMRegistry extends AMRegistry { + + private static final Logger LOG = LoggerFactory.getLogger(ZkAMRegistry.class); + + private final List amRecords = new ArrayList<>(); + private final String externalId; + + private CuratorFramework client = null; + private String namespace = null; + private ZkConfig zkConfig = null; + private boolean started = false; + + public ZkAMRegistry(String externalId) { +super("ZkAMRegistry"); +this.externalId = externalId; + } + + @Override + public void serviceInit(Configuration conf) { +if (zkConfig == null) { + zkConfig = new ZkConfig(conf); + this.client = zkConfig.createCuratorFramework(); + this.namespace = zkConfig.getZkNamespace(); + LOG.info("AMRegistryZkImpl initialized"); +} + } + + @Override + public void serviceStart() throws Exception { +if (!started) { + client.start(); + started = true; + LOG.info("AMRegistryZkImpl started"); +} + } + + //Deletes from Zookeeper AMRecords that were added by this instance + @Override + public void serviceStop() throws Exception { +List records = new ArrayList<>(amRecords); +for (AMRecord amRecord : records) { + remove(amRecord); +} +client.close(); +LOG.info("AMRegistryZkImpl shutdown"); + } + + //Serialize AMRecord to ServiceRecord and deliver the JSON bytes to + //zkNode at the path: / + @Override + public void add(AMRecord server) throws Exception { +RegistryUtils.ServiceRecordMarshal marshal = new RegistryUtils.ServiceRecordMarshal(); +String json = marshal.toJson(server.toServiceRecord()); +try { + final String path = namespace + "/" + server.getApplicationId().toString(); + client.setData().forPath(path, json.getBytes(StandardCharsets.UTF_8)); + LOG.info("Added AMRecord to zkpath {}", path); +} catch (KeeperException.NoNodeException nne) { + client.create().creatingParentContainersIfNeeded().withMode(CreateMode.EPHEMERAL) Review Comment: EPHEMERAL looks good important difference between EPHEMERAL and PERSISTENT is that: ``` /** * The znode will be deleted upon the client's disconnect. */ EPHEMERAL(1, true, false, false, false), ``` vs ``` /** * The znode will not be automatically deleted upon client's disconnect. */ PERSISTENT(0, false, false, false, false), ``` in our usecase, an AM registers itself to ZK, so when AM is about to shutdown and client disconnects, it sounds quite convenient to get the corresponding nodes automatically deleted (node deletion -> update on client side about AM has been removed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2565154649
##
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java:
##
@@ -1612,19 +1613,14 @@ DAGState initializeDAG() {
addVertex(v);
}
-// check task resources, only check it in non-local mode
if (!appContext.isLocal()) {
- for (Vertex v : vertexMap.values()) {
-// TODO TEZ-2003 (post) TEZ-2624 Ideally, this should be per source.
-if
(v.getTaskResource().compareTo(appContext.getClusterInfo().getMaxContainerCapability())
> 0) {
- String msg = "Vertex's TaskResource is beyond the cluster container
capability," +
- "Vertex=" + v.getLogIdentifier() +", Requested TaskResource=" +
v.getTaskResource()
- + ", Cluster MaxContainerCapability=" +
appContext.getClusterInfo().getMaxContainerCapability();
- LOG.error(msg);
- addDiagnostic(msg);
- finished(DAGState.FAILED);
- return DAGState.FAILED;
-}
+ try {
+appContext.getAmExtensions().checkTaskResources(vertexMap,
appContext.getClusterInfo());
+ } catch (TaskResourceException e) {
Review Comment:
ack, also the specialized TaskResourceException exception can be removed, it
has no added value at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2565140913
##
tez-api/src/test/java/org/apache/tez/client/registry/zookeeper/TestZkFrameworkClient.java:
##
@@ -0,0 +1,243 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.zookeeper.CreateMode;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Unit tests for {@link ZkFrameworkClient}.
+ *
+ * This test class validates the ZooKeeper-based framework client that
discovers
+ * and communicates with Application Masters through ZooKeeper registry.
+ *
+ */
+public class TestZkFrameworkClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(TestZkFrameworkClient.class);
+ private static final File TEST_DIR = new
File(System.getProperty("test.build.data", "target"),
+ TestZkFrameworkClient.class.getName()).getAbsoluteFile();
+
+ private TestingServer zkServer;
+ private ZkFrameworkClient zkFrameworkClient;
+ private CuratorFramework curatorClient;
+
+ @Before
+ public void setup() throws Exception {
+zkServer = new TestingServer(true);
+LOG.info("Started ZooKeeper test server on port: {}", zkServer.getPort());
+ }
+
+ @After
+ public void teardown() throws Exception {
+if (zkFrameworkClient != null) {
+ zkFrameworkClient.close();
+}
+IOUtils.closeQuietly(curatorClient);
+IOUtils.closeQuietly(zkServer);
+ }
+
+ /**
+ * Tests initialization and lifecycle methods of ZkFrameworkClient.
+ */
+ @Test
+ public void testInitAndLifecycle() throws Exception {
+TezConfiguration tezConf = createTezConf();
+
+zkFrameworkClient = new ZkFrameworkClient();
+zkFrameworkClient.init(tezConf);
+
+assertTrue("Client should be running after init",
zkFrameworkClient.isRunning());
+
+zkFrameworkClient.start();
+assertTrue("Client should be running after start",
zkFrameworkClient.isRunning());
+
+zkFrameworkClient.stop();
+assertFalse("Client should not be running after stop",
zkFrameworkClient.isRunning());
+ }
+
+ /**
+ * Tests retrieving application report when AM is registered in ZooKeeper.
+ */
+ @Test
+ public void testGetApplicationReportWithRegisteredAM() throws Exception {
+TezConfiguration tezConf = createTezConf();
+
+// Register a mock AM in ZooKeeper
+ApplicationId appId =
ApplicationId.newInstance(System.currentTimeMillis(), 1);
+String testHostName = "test-host";
+String testHostIp = "127.0.0.1";
+int testPort = 12345;
+registerMockAM(tezConf, appId, testHostName, testHostIp, testPort);
+
+zkFrameworkClient = new ZkFrameworkClient();
+zkFrameworkClient.init(tezConf);
+zkFrameworkClient.start();
+
+// Give time for ZK registry to initialize
+Thread.sleep(500);
Review Comment:
ack, the ZK TreeCache INITIALIZED callback can be used for such purposes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2565040247
##
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
##
@@ -447,6 +457,94 @@ public TezClientForTest testTezClient(boolean isSession,
boolean shouldStop, Str
return client;
}
+ public void testTezClientReconnect(boolean isSession) throws Exception {
+//Setup 1
+Map lrs = Maps.newHashMap();
+String lrName1 = "LR1";
+lrs.put(lrName1, LocalResource.newInstance(URL.newInstance("file",
"localhost", 0, "/test"),
+LocalResourceType.FILE, LocalResourceVisibility.PUBLIC, 1, 1));
+
+//Client 1
+TezClientForTest client = configureAndCreateTezClient(lrs, isSession,
null);
+
+//Submission Context 1
+ArgumentCaptor captor =
ArgumentCaptor.forClass(ApplicationSubmissionContext.class);
+
when(client.mockYarnClient.getApplicationReport(client.mockAppId).getYarnApplicationState())
+.thenReturn(YarnApplicationState.RUNNING);
+
+//Client 1 start
+client.start();
Review Comment:
yeah, dagClient can and should be closed
I cannot see separate close method in TezClient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2564918638
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
finally, I decided to remove getByServiceLoader, and make the users use it
by getMode properly
let's keep it simple
also FrameworkUtils.get can be provided with a default class parameter to
handle this case easily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2564918638
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
finally, I decided to remove getByServiceLoader, and make the users use it
by getMode properly
let's keep it simple
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563630356
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
**UPDATE (ignore things below): I just realized this is already handled in
the `getByMode` before the `getByServiceLoader`, so this is another case, sorry
for the noise, let me check this**
yeah, I was thinking about the same, which might look hacky, but maybe it's
worth it
the problem is that this method is in `tez-api`, whereas the server-side
stuff is in `tez-dag`, and `tez-api` by design cannot depend on `tez-dag`, so
instead of the null-checks, this might need at least two runtime workarounds:
1. check if `interfaze` is `ServerFrameworkService` or
`ClientFrameworkService`, and return Yarn one accordingly
2. instantiate the `YarnServerFrameworkService` by reflection (reflection is
needed here, because even if moving the `ServerFrameworkService` to `tez-api`
fine, doing the same with `YarnServerFrameworkService` looks strange, as it's
Tez AM specific implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563630356
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
yeah, I was thinking about the same, which might look hacky, but maybe it's
worth it
the problem is that this method is in `tez-api`, whereas the server-side
stuff is in `tez-dag`, and `tez-api` by design cannot depend on `tez-dag`, so
instead of the null-checks, this might need at least two runtime workarounds:
1. check if `interfaze` is `ServerFrameworkService` or
`ClientFrameworkService`, and return Yarn one accordingly
2. instantiate the `YarnServerFrameworkService` by reflection (reflection is
needed here, because even if moving the `ServerFrameworkService` to `tez-api`
fine, doing the same with `YarnServerFrameworkService` looks strange, as it's
Tez AM specific implementation
UPDATE: I just realized this is already handled in the `getByMode` before
the `getByServiceLoader`, so this is another case, sorry for the noise, let me
check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563630356
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
yeah, I was thinking about the same, which might look hacky, but maybe it's
worth it
the problem is that this method is in `tez-api`, whereas the server-side
stuff is in `tez-dag`, and `tez-api` by design cannot depend on `tez-dag`, so
instead of the null-checks, this might need at least two runtime workarounds:
1. check if `interfaze` is `ServerFrameworkService` or
`ClientFrameworkService`, and return Yarn one accordingly
2. instantiate the `YarnServerFrameworkService` by reflection (reflection is
needed here, because even if moving the `ServerFrameworkService` to `tez-api`
fine, doing the same with `YarnServerFrameworkService` looks strange, as it's
Tez AM specific implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563630356
##
tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java:
##
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.ServiceLoader;
+
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.common.ReflectionUtils;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.TezReflectionException;
+
+public final class FrameworkUtils {
+
+ private FrameworkUtils() {}
+
+ /*
+Searches for a FrameworkService provider which implements a target
interface.
+The interface should be either ClientFrameworkService or
ServerFrameworkService.
+Depending on which interface is used, either the client or server class of
a
+matching FrameworkMode will be used as the implementation.
+
+NOTE: Layering of FrameworkServices in a decorator-style is currently not
supported
+
+An implementation is searched in the following order:
+ 1. If conf is not null and the parameter TEZ_FRAMEWORK_MODE is set:
+ the value of TEZ_FRAMEWORK_MODE from the conf will be used
+ 2. If conf is null or the parameter TEZ_FRAMEWORK_MODE is not set
+and the environment var TEZ_FRAMEWORK_MODE is not empty:
+the value of the environment var will be used
+ 3. Otherwise:
+ the default java.util.ServiceLoader behavior will be used,
+ i.e. the implementation classname should appear in a file on the
classpath at the location
+
META-INF/services/org.apache.tez.frameworkplugins.ClientFrameworkService
+or
META-INF/services/org.apache.tez.frameworkplugins.ServerFrameworkService
+ */
+ public static T get(Class interfaze,
@Nullable Configuration conf) {
+try {
+ if ((conf != null) && (conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) !=
null)) {
+return getByMode(interfaze,
conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE));
+ } else if (System.getenv(TezConstants.TEZ_FRAMEWORK_MODE) != null) {
+return getByMode(interfaze,
System.getenv(TezConstants.TEZ_FRAMEWORK_MODE));
+ } else {
+return getByServiceLoader(interfaze);
+ }
+} catch (TezReflectionException e) {
+ throw new RuntimeException("Failed to load framework service for
interface: " + interfaze.getName(), e);
+}
+ }
+
+ private static T getByServiceLoader(Class
interfaze) {
+List services = new ArrayList<>();
+ServiceLoader frameworkService = ServiceLoader.load(interfaze);
+for (T service : frameworkService) {
+ services.add(service);
+}
+if (services.isEmpty()) {
+ return null;
Review Comment:
yeah, I was thinking about the same, which might look hacky, but maybe it's
worth it
this problem is that this method is in tez-api, whereas the server side
stuff is in tez-dag, and tez-api by design cannot depend on tez-dag, so instead
of the null-checks, this might need at least 2 runtime workarounds:
1. check if interfaze is `ServerFrameworkService` or
`ClientFrameworkService`, and return Yarn one accordingly
2. instantiate the `YarnServerFrameworkService` by reflection (reflection is
needed here, becuase even if moving the `ServerFrameworkService` to tez-api
fine, doing the same with `YarnServerFrameworkService` looks strange, as it's
Tez AM specific implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563596644
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkConfig.java:
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ZkConfig {
+ private static final Logger LOG = LoggerFactory.getLogger(ZkConfig.class);
+
+ // if namespace defined in config is 'foo' and COMPUTE_GROUP_NAME env is
'bar' then the zkpaths will be of format
+ // /tez-external-sessions/foo/bar
+ private final static String ZK_NAMESPACE_PREFIX = "/tez-external-sessions";
+ public final static String COMPUTE_GROUP_NAME_ENV = "COMPUTE_GROUP_NAME";
+ public final static String DEFAULT_COMPUTE_GROUP_NAME = "default-compute";
+
+ private final String zkQuorum;
+ private final String zkNamespace;
+ private final int curatorBackoffSleep;
+ private final int curatorMaxRetries;
+ private final int sessionTimeoutMs;
+ private final int connectionTimeoutMs;
+
+ public ZkConfig(Configuration conf) {
+zkQuorum = conf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+Preconditions.checkNotNull(zkQuorum);
+
+String fullZkNamespace = ZK_NAMESPACE_PREFIX;
+
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE,
+TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE_DEFAULT);
+Preconditions.checkNotNull(namespace);
+if (namespace.startsWith(Path.SEPARATOR)) {
+ fullZkNamespace += namespace;
+} else {
+ fullZkNamespace = fullZkNamespace + Path.SEPARATOR + namespace;
+}
+
+boolean enableComputeGroups =
conf.getBoolean(TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS,
+TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS_DEFAULT);
+if (enableComputeGroups) {
+ final String subNamespace = System.getenv(COMPUTE_GROUP_NAME_ENV);
Review Comment:
this regex replaces leading slashes with an empty string, which leads to
this exception in ZK:
`java.lang.IllegalArgumentException: Path must start with / character`
in ZK, a node should start with a slash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2563595946
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkConfig.java:
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ZkConfig {
+ private static final Logger LOG = LoggerFactory.getLogger(ZkConfig.class);
+
+ // if namespace defined in config is 'foo' and COMPUTE_GROUP_NAME env is
'bar' then the zkpaths will be of format
+ // /tez-external-sessions/foo/bar
+ private final static String ZK_NAMESPACE_PREFIX = "/tez-external-sessions";
+ public final static String COMPUTE_GROUP_NAME_ENV = "COMPUTE_GROUP_NAME";
+ public final static String DEFAULT_COMPUTE_GROUP_NAME = "default-compute";
+
+ private final String zkQuorum;
+ private final String zkNamespace;
+ private final int curatorBackoffSleep;
+ private final int curatorMaxRetries;
+ private final int sessionTimeoutMs;
+ private final int connectionTimeoutMs;
+
+ public ZkConfig(Configuration conf) {
+zkQuorum = conf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+Preconditions.checkNotNull(zkQuorum);
+
+String fullZkNamespace = ZK_NAMESPACE_PREFIX;
+
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE,
Review Comment:
this regex replaces leading slashes with an empty string, which leads to
this exception in ZK:
`java.lang.IllegalArgumentException: Path must start with / character`
in ZK, a node should start with a slash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2561667762
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
+ private ZkAMRegistryClient amRegistryClient = null;
+ private volatile boolean isRunning = false;
+ private String amHost;
+ private int amPort;
+
+ @Override
+ public synchronized void init(TezConfiguration tezConf) {
+if (this.amRegistryClient == null) {
+ try {
+this.amRegistryClient = ZkAMRegistryClient.getClient(tezConf);
+this.isRunning = true;
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ this.amRegistryClient.start();
+ isRunning = true;
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ @Override
+ public void stop() {
+isRunning = false;
+close();
+ }
+
+ @Override
+ public void close() {
+if (amRegistryClient != null) {
+ amRegistryClient.close();
+}
+ }
+
+ /**
+ * Creates a dummy {@link YarnClientApplication} using a pre-existing {@link
ApplicationId}
+ * rather than requesting a new one from the ResourceManager.
+ *
+ * Note: This is a dummy, backward-compatibility
implementation.
+ * Instead of allocating a fresh application ID from the ResourceManager,
this method
+ * reuses the {@code applicationId} already obtained via {@code
getApplicationReport()}.
+ * This allows legacy code paths to continue operating without requiring
actual
+ * creation of a new application.
+ *
+ * Hidden assumption here: this method assumes that
+ * {@code getApplicationReport()} has already been called before
+ * {@code createApplication()}, ensuring that {@code
amRecord.getApplicationId()}
+ * is always available. This assumption holds in all supported usage
patterns:
+ * the only code path where {@code createApplication()} might be called
first is
+ * {@code TezClient.submitDAGApplication()}, but that path is never
exercised in
+ * Zookeeper standalone mode because that mode assumes applications are
already
+ * running. Therefore, the ordering guarantee is valid in practice.
+ *
+ *
+ * The method constructs a minimal {@link ApplicationSubmissionContext} and a
+ * synthetic {@link GetNewApplicationResponse}, both populated with the
already
+ * known application ID. These objects are then wrapped into a
+ * {@link YarnClientApplication} instance and returned.
+ *
+ *
+ * @return a {@link YarnClientApplication} backed by a submission context and
+ * a mocked {@link GetNewApplicationResponse}, both tied to the
pre-existing
+ * application ID.
+ */
+ @Override
+ public YarnClientApplication createApplication() {
+ApplicationSubmissionContext context =
Records.newRecord(ApplicationSubmissionContext.class);
+ApplicationId appId = amRecord.getApplicationId();
Review Comment:
I know it looks confusing, and already added a javadoc explain this:
```
* Note: This is a dummy, backward-compatibility
implementation.
* Instead of allocating a fresh application ID from the ResourceManager,
this
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2561657222
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkConfig.java:
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ZkConfig {
+ private static final Logger LOG = LoggerFactory.getLogger(ZkConfig.class);
+
+ // if namespace defined in config is 'foo' and COMPUTE_GROUP_NAME env is
'bar' then the zkpaths will be of format
+ // /tez-external-sessions/foo/bar
+ private final static String ZK_NAMESPACE_PREFIX = "/tez-external-sessions";
+ public final static String COMPUTE_GROUP_NAME_ENV = "COMPUTE_GROUP_NAME";
+ public final static String DEFAULT_COMPUTE_GROUP_NAME = "default-compute";
+
+ private final String zkQuorum;
+ private final String zkNamespace;
+ private final int curatorBackoffSleep;
+ private final int curatorMaxRetries;
+ private final int sessionTimeoutMs;
+ private final int connectionTimeoutMs;
+
+ public ZkConfig(Configuration conf) {
+zkQuorum = conf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+Preconditions.checkNotNull(zkQuorum);
+
+String fullZkNamespace = ZK_NAMESPACE_PREFIX;
+
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE,
+TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE_DEFAULT);
+Preconditions.checkNotNull(namespace);
+if (namespace.startsWith(Path.SEPARATOR)) {
+ fullZkNamespace += namespace;
+} else {
+ fullZkNamespace = fullZkNamespace + Path.SEPARATOR + namespace;
+}
+
+boolean enableComputeGroups =
conf.getBoolean(TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS,
+TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS_DEFAULT);
+if (enableComputeGroups) {
+ final String subNamespace = System.getenv(COMPUTE_GROUP_NAME_ENV);
+ if (subNamespace != null && !subNamespace.isEmpty()) {
+if (subNamespace.startsWith(Path.SEPARATOR)) {
+ fullZkNamespace += subNamespace;
+} else {
+ fullZkNamespace = fullZkNamespace + Path.SEPARATOR + subNamespace;
+}
+LOG.info("Compute groups enabled: subNamespace: {} fullZkNamespace:
{}", subNamespace, fullZkNamespace);
+ }
+} else {
+ LOG.info("Compute groups disabled: fullZkNamespace: {}",
fullZkNamespace);
+}
+zkNamespace = fullZkNamespace;
+
+curatorBackoffSleep =
conf.getInt(TezConfiguration.TEZ_AM_CURATOR_BACKOFF_SLEEP,
+TezConfiguration.TEZ_AM_CURATOR_BACKOFF_SLEEP_DEFAULT);
+curatorMaxRetries =
conf.getInt(TezConfiguration.TEZ_AM_CURATOR_MAX_RETRIES,
+TezConfiguration.TEZ_AM_CURATOR_MAX_RETRIES_DEFAULT);
+sessionTimeoutMs =
conf.getInt(TezConfiguration.TEZ_AM_CURATOR_SESSION_TIMEOUT,
+TezConfiguration.TEZ_AM_CURATOR_SESSION_TIMEOUT_DEFAULT);
+connectionTimeoutMs =
conf.getInt(TezConfiguration.TEZ_AM_CURATOR_CONNECTION_TIMEOUT,
+TezConfiguration.TEZ_AM_CURATOR_CONNECTION_TIMEOUT_DEFAULT);
+ }
+
+ public String getZkQuorum() {
+return zkQuorum;
+ }
+
+ public String getZkNamespace() {
+return zkNamespace;
+ }
+
+ public int getCuratorBackoffSleep() {
+return curatorBackoffSleep;
+ }
+
+ public int getCuratorMaxRetries() {
+return curatorMaxRetries;
+ }
+
+ public int getSessionTimeoutMs() {
+return sessionTimeoutMs;
+ }
+
+ public int getConnectionTimeoutMs() {
+return connectionTimeoutMs;
+ }
+
+ public RetryPolicy getRetryPolicy() {
+return new ExponentialBackoffRetry(getCuratorBackoffSleep(),
getCuratorMaxRetries());
+ }
+
+ public CuratorFramework createCuratorFramework() {
Review Comment:
yeah, close is taken care of n
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
okumin commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2552095265
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
+ private ZkAMRegistryClient amRegistryClient = null;
+ private volatile boolean isRunning = false;
+ private String amHost;
+ private int amPort;
+
+ @Override
+ public synchronized void init(TezConfiguration tezConf) {
+if (this.amRegistryClient == null) {
+ try {
+this.amRegistryClient = ZkAMRegistryClient.getClient(tezConf);
+this.isRunning = true;
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ this.amRegistryClient.start();
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+isRunning = true;
+ }
+
+ @Override
+ public void stop() {
+isRunning = false;
+amRegistryClient.close();
+ }
+
+ @Override
+ public void close() throws IOException {
+amRegistryClient.close();
+ }
+
+ @Override
+ public YarnClientApplication createApplication() throws YarnException,
IOException {
+ApplicationSubmissionContext context =
Records.newRecord(ApplicationSubmissionContext.class);
+ApplicationId appId = amRecord.getApplicationId();
Review Comment:
Thanks. I presume that, on YARN with the session mode, Tez creates an AM
when a `TezClient` starts. In the case of ZK Standalone, I'm guessing an AM is
made outside the lifecycle of a `TezClient`, and the TezClient pulls an app
with an explicitly passed app ID. I'm still not 100% confident, and I'd
appreciate it if you could give me tips. I put my understanding below.
https://docs.google.com/document/d/1SGIwfXgfwMfrtrHvNwdLsdbbHkb--5BsGJ9yt-9-bP0/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2551051434
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2551047023
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2551031624
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549826279
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549826279
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549815047
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549815047
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java:
##
@@ -0,0 +1,205 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.shaded.com.google.common.base.Charsets;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.registry.client.binding.RegistryUtils;
+import org.apache.hadoop.registry.client.types.ServiceRecord;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.client.registry.AMRegistryClient;
+import org.apache.tez.client.registry.AMRegistryClientListener;
+import org.apache.tez.dag.api.TezConfiguration;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.google.common.base.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Curator/Zookeeper implementation of {@link AMRegistryClient}.
+ */
[email protected]
+public final class ZkAMRegistryClient extends AMRegistryClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZkAMRegistryClient.class);
+ private static final Map INSTANCES = new
HashMap<>();
+
+ private final Configuration conf;
+ //Cache of known AMs
+ private final ConcurrentHashMap amRecordCache = new
ConcurrentHashMap<>();
+ private CuratorFramework client;
+
+ private ZkAMRegistryClient(final Configuration conf) {
+this.conf = conf;
+ }
+
+ public static synchronized ZkAMRegistryClient getClient(final Configuration
conf) {
+String namespace = conf.get(TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE);
+ZkAMRegistryClient registry = INSTANCES.get(namespace);
+if (registry == null) {
+ registry = new ZkAMRegistryClient(conf);
+ INSTANCES.put(namespace, registry);
+}
+LOG.info("Returning tez AM registry ({}) for namespace '{}'",
System.identityHashCode(registry), namespace);
+return registry;
+ }
+
+ /**
+ * Deserializes a {@link ServiceRecord} from ZooKeeper data and converts it
into an {@link AMRecord}
+ * for caching.
+ *
+ * @param childData the ZooKeeper node data containing a serialized {@link
ServiceRecord}
+ * @return an {@link AMRecord} constructed from the deserialized {@link
ServiceRecord}, or {@code null}
+ * if no data is present
+ * @throws IOException if the data cannot be deserialized into a {@link
ServiceRecord}
+ */
+ public static AMRecord getAMRecord(final ChildData childData) throws
IOException {
+// not a leaf path. Only leaf path contains AMRecord
+if (!childData.getPath().contains(ApplicationId.appIdStrPrefix)) {
+ return null;
+}
+byte[] data = childData.getData();
+// only the path appeared, there is no data yet
+if (data.length == 0) {
+ return null;
+}
+String value = new String(data, Charsets.UTF_8);
+try {
+ RegistryUtils.ServiceRecordMarshal marshal = new
RegistryUtils.ServiceRecordMarshal();
+ ServiceRecord serviceRecord = marshal.fromJson(value);
+ return new AMRecord(serviceRecord);
+} catch (JsonParseException e) {
+ //Not a json AMRecord (SRV), could be some other data.
+ LOG.warn("Non-json data received while de-serializing AMRecord: {}.
Ignoring...", value);
+ return null;
+}
+ }
+
+ public void start() throws Exception {
+ZkConfig zkConf = new ZkConfig(this.conf);
+client = zkConf.createCuratorFram
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549763920
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Returns the current set of registered listeners.
+ *
+ * @return a mutable list of listeners
+ */
+ protected List getListeners() {
+return listeners;
+ }
+
+ /**
+ * Lookup AM metadata for the given application ID.
+ *
+ * @param appId the application ID
+ * @return the AM record if found, otherwise {@code null}
+ * @throws IOException if the lookup fails
+ */
+ public abstract AMRecord getRecord(String appId) throws IOException;
Review Comment:
yeah, also, as ApplicationId's equals and hashCode are properly implemented,
we can use it as the Map key type also in ZkAMRegistryClient
```
private final ConcurrentHashMap amRecordCache =
new ConcurrentHashMap<>();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549763920
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Returns the current set of registered listeners.
+ *
+ * @return a mutable list of listeners
+ */
+ protected List getListeners() {
+return listeners;
+ }
+
+ /**
+ * Lookup AM metadata for the given application ID.
+ *
+ * @param appId the application ID
+ * @return the AM record if found, otherwise {@code null}
+ * @throws IOException if the lookup fails
+ */
+ public abstract AMRecord getRecord(String appId) throws IOException;
Review Comment:
yeah, also, as ApplicationId's equals and hashCode are properly implemented,
we can use it as the Map key type also
```
private final ConcurrentHashMap amRecordCache =
new ConcurrentHashMap<>();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549747379
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Returns the current set of registered listeners.
+ *
+ * @return a mutable list of listeners
+ */
+ protected List getListeners() {
+return listeners;
+ }
Review Comment:
ack, removing getListeners altogether
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549739970
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java:
##
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Client-side interface for discovering Application Master (AM) instances
+ * registered in the AM registry.
+ *
+ * Implementations are responsible for locating AM endpoints and returning
+ * their metadata. This API is used by client components to discover running
+ * Tez AMs.
+ *
+ * Listeners may be registered to receive notifications when AM records
+ * appear or are removed.
+ */
+public abstract class AMRegistryClient implements Closeable {
+
+ private final List listeners = new ArrayList<>();
+
+ /**
+ * Returns the current set of registered listeners.
+ *
+ * @return a mutable list of listeners
+ */
+ protected List getListeners() {
+return listeners;
+ }
+
+ /**
+ * Lookup AM metadata for the given application ID.
+ *
+ * @param appId the application ID
+ * @return the AM record if found, otherwise {@code null}
+ * @throws IOException if the lookup fails
+ */
+ public abstract AMRecord getRecord(String appId) throws IOException;
+
+ /**
+ * Retrieve all AM records known in the registry.
+ *
+ * @return a list of AM records (possibly empty)
+ * @throws IOException if the fetch fails
+ */
+ public abstract List getAllRecords() throws IOException;
+
+ /**
+ * Register a listener for AM registry events.
+ * The listener will be notified when AM records are added or removed.
+ *
+ * @param listener the listener to add
+ */
+ public synchronized void addListener(AMRegistryClientListener listener) {
+listeners.add(listener);
+ }
+
+ /**
+ * Notify listeners of a newly added AM record.
+ *
+ * @param record the added AM record
+ */
+ protected synchronized void notifyOnAdded(AMRecord record) {
+for (AMRegistryClientListener listener : listeners) {
+ listener.onAdd(record);
+}
Review Comment:
yeah, same applies to removal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549694323
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java:
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.service.AbstractService;
+import org.apache.hadoop.service.ServiceStateException;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for {@code AMRegistry} implementations.
+ *
+ * The specific implementation is configured via the
+ * {@code tez.am.registry.class} property.
+ *
+ * Implementations are expected to provide appropriate service lifecycle
+ * behavior, including:
+ *
+ * {@code init}
+ * {@code serviceStart}
+ * {@code serviceStop}
+ *
+ *
+ *
+ * {@code init} and {@code serviceStart} are invoked during
+ * {@code DAGAppMaster.serviceInit()}, while {@code serviceStop} is called
+ * when {@code DAGAppMaster} shuts down.
+ */
+public abstract class AMRegistry extends AbstractService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AMRegistry.class);
+ private List amRecords = new ArrayList<>();
Review Comment:
also, there is a private amRecords in the ZkAMRegistry subclass, which makes
it really confusing, let me cleanup this codepath
EDIT: moving the collection to the subclass, making a final, synchronized
collection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549694323
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java:
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.service.AbstractService;
+import org.apache.hadoop.service.ServiceStateException;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for {@code AMRegistry} implementations.
+ *
+ * The specific implementation is configured via the
+ * {@code tez.am.registry.class} property.
+ *
+ * Implementations are expected to provide appropriate service lifecycle
+ * behavior, including:
+ *
+ * {@code init}
+ * {@code serviceStart}
+ * {@code serviceStop}
+ *
+ *
+ *
+ * {@code init} and {@code serviceStart} are invoked during
+ * {@code DAGAppMaster.serviceInit()}, while {@code serviceStop} is called
+ * when {@code DAGAppMaster} shuts down.
+ */
+public abstract class AMRegistry extends AbstractService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AMRegistry.class);
+ private List amRecords = new ArrayList<>();
Review Comment:
also, there is a private amRecords in the ZkAMRegistry subclass, which makes
it really confusing, let me cleanup this codepath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549684127
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java:
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.service.AbstractService;
+import org.apache.hadoop.service.ServiceStateException;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for {@code AMRegistry} implementations.
+ *
+ * The specific implementation is configured via the
+ * {@code tez.am.registry.class} property.
+ *
+ * Implementations are expected to provide appropriate service lifecycle
+ * behavior, including:
+ *
+ * {@code init}
+ * {@code serviceStart}
+ * {@code serviceStop}
+ *
+ *
+ *
+ * {@code init} and {@code serviceStart} are invoked during
+ * {@code DAGAppMaster.serviceInit()}, while {@code serviceStop} is called
+ * when {@code DAGAppMaster} shuts down.
+ */
+public abstract class AMRegistry extends AbstractService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AMRegistry.class);
+ private List amRecords = new ArrayList<>();
+
+ @Override
+ public void init(Configuration conf) {
+try {
+ this.serviceInit(conf);
+} catch (Exception e) {
+ LOG.error("Failed to init AMRegistry: name={}, type={}", getName(),
getClass().getName());
+ throw ServiceStateException.convert(e);
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ this.serviceStart();
+} catch(Exception e) {
+ LOG.error("Failed to start AMRegistry: name={}, type={}", getName(),
getClass().getName());
+ throw ServiceStateException.convert(e);
+}
+ }
Review Comment:
right, converting the exceptions to ServiceStateException doesn't add
obvious value here anyway, I'm removing these overrides
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549661395
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -150,14 +180,21 @@ public boolean equals(Object other) {
public ServiceRecord toServiceRecord() {
ServiceRecord serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
-serviceRecord.set(HOST_RECORD_KEY, host);
+serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
+serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
-serviceRecord.set(OPAQUE_ID_KEY, id);
+serviceRecord.set(EXTERNAL_ID_KEY, externalId);
+serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
return serviceRecord;
}
+ @Override
+ public String toString() {
+return toServiceRecord().attributes().toString();
+ }
Review Comment:
used in INFO level messages in downstream-only tez session client
implementation, and toServiceRecord itself is used in ZkAMRegistry to store AM
information in zookeeper as JSON
would you be fine to cache ServiceRecord in a field and return it like:
```
public ServiceRecord toServiceRecord() {
if (serviceRecord != null) {
return serviceRecord;
}
serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
serviceRecord.set(EXTERNAL_ID_KEY, externalId);
serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
return serviceRecord;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549661395
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -150,14 +180,21 @@ public boolean equals(Object other) {
public ServiceRecord toServiceRecord() {
ServiceRecord serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
-serviceRecord.set(HOST_RECORD_KEY, host);
+serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
+serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
-serviceRecord.set(OPAQUE_ID_KEY, id);
+serviceRecord.set(EXTERNAL_ID_KEY, externalId);
+serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
return serviceRecord;
}
+ @Override
+ public String toString() {
+return toServiceRecord().attributes().toString();
+ }
Review Comment:
used in INFO level messages in downstream-only tez session client
implementation, and toServiceRecord itself is used in ZkAMRegistry to store AM
information in zookeeper as JSON
would you be fine to cache ServiceRecord in a field and return it like:
```
public ServiceRecord toServiceRecord() {
if (serviceRecord != null){
return serviceRecord;
}
serviceRecord = new ServiceRecord();
serviceRecord.set(APP_ID_RECORD_KEY, appId);
serviceRecord.set(HOST_NAME_RECORD_KEY, hostName);
serviceRecord.set(HOST_IP_RECORD_KEY, hostIp);
serviceRecord.set(PORT_RECORD_KEY, port);
serviceRecord.set(EXTERNAL_ID_KEY, externalId);
serviceRecord.set(COMPUTE_GROUP_NAME_KEY, computeName);
return serviceRecord;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549593120
##
tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java:
##
@@ -97,37 +110,54 @@ public AMRecord(AMRecord other) {
*/
public AMRecord(ServiceRecord serviceRecord) {
this.appId =
ApplicationId.fromString(serviceRecord.get(APP_ID_RECORD_KEY));
-this.host = serviceRecord.get(HOST_RECORD_KEY);
+this.hostName = serviceRecord.get(HOST_NAME_RECORD_KEY);
+this.hostIp = serviceRecord.get(HOST_IP_RECORD_KEY);
this.port = Integer.parseInt(serviceRecord.get(PORT_RECORD_KEY));
-this.id = serviceRecord.get(OPAQUE_ID_KEY);
+this.externalId = serviceRecord.get(EXTERNAL_ID_KEY);
+this.computeName = serviceRecord.get(COMPUTE_GROUP_NAME_KEY);
}
public ApplicationId getApplicationId() {
return appId;
}
public String getHost() {
-return host;
+return hostName;
+ }
+
+ public String getHostName() {
+return hostName;
+ }
Review Comment:
good catch! let's have it as `getHostName` to make a clear distinction
between this and `getHostIp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2549577799
##
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
##
@@ -484,7 +484,9 @@ public synchronized TezClient getClient(ApplicationId
appId) throws TezException
}
private void startFrameworkClient() {
-frameworkClient = createFrameworkClient();
+if (frameworkClient == null) {
+ frameworkClient = createFrameworkClient();
+}
Review Comment:
right, if we want to make it idempotent, it should instead be:
```
if (frameworkClient != null) {
return;
}
frameworkClient = createFrameworkClient();
frameworkClient.init(amConfig.getTezConfiguration());
frameworkClient.start();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3562298246
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 11s | | master passed |
| +1 :green_heart: | compile | 5m 18s | | master passed |
| +1 :green_heart: | checkstyle | 3m 2s | | master passed |
| +1 :green_heart: | javadoc | 3m 53s | | master passed |
| +0 :ok: | spotbugs | 2m 12s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 46s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 12s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 44s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 45s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 31s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 8m 14s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 46s | | the patch passed |
| +1 :green_heart: | codespell | 0m 55s | | No new issues. |
| +1 :green_heart: | compile | 5m 22s | | the patch passed |
| +1 :green_heart: | javac | 5m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 17s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 26s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 20s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/26/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 42s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/26/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 40s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 35s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 34s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 41s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 5m 58s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 31s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 5m 26s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 71m 57s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 41s | | The patch does not
generate ASF License warnings. |
| | | 168m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/26/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux 7640fc86df1a 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3558453040
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 15s | | master passed |
| +1 :green_heart: | compile | 5m 1s | | master passed |
| +1 :green_heart: | checkstyle | 2m 51s | | master passed |
| +1 :green_heart: | javadoc | 3m 38s | | master passed |
| +0 :ok: | spotbugs | 2m 6s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 41s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 8s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 40s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 41s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 6m 54s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 37s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 38s | | the patch passed |
| +1 :green_heart: | codespell | 0m 51s | | No new issues. |
| +1 :green_heart: | compile | 5m 6s | | the patch passed |
| +1 :green_heart: | javac | 5m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 22s |
[/results-checkstyle-tez-api.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/25/artifact/out/results-checkstyle-tez-api.txt)
| tez-api: The patch generated 5 new + 118 unchanged - 3 fixed = 123 total
(was 121) |
| +1 :green_heart: | checkstyle | 0m 19s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 18s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/25/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 40s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/25/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 740 unchanged - 6 fixed = 746 total (was
746) |
| +1 :green_heart: | javadoc | 3m 30s | | the patch passed |
| -1 :x: | spotbugs | 1m 41s |
[/new-spotbugs-tez-api.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/25/artifact/out/new-spotbugs-tez-api.html)
| tez-api generated 1 new + 610 unchanged - 0 fixed = 611 total (was 610) |
| -1 :x: | spotbugs | 7m 19s |
[/new-spotbugs-root.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/25/artifact/out/new-spotbugs-root.html)
| root generated 1 new + 2066 unchanged - 0 fixed = 2067 total (was 2066) |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 30s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 6m 1s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 28s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 5m 20s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 72m 45s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 13s | | The patch does not
generate ASF License warnings. |
| | | 166m 3s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:tez-api |
| |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3557195271
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 19s | | master passed |
| +1 :green_heart: | compile | 4m 58s | | master passed |
| +1 :green_heart: | checkstyle | 2m 49s | | master passed |
| +1 :green_heart: | javadoc | 3m 39s | | master passed |
| +0 :ok: | spotbugs | 2m 9s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 42s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 8s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 40s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 37s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 7m 0s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 43s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 32s | | the patch passed |
| +1 :green_heart: | codespell | 0m 56s | | No new issues. |
| +1 :green_heart: | compile | 5m 2s | | the patch passed |
| +1 :green_heart: | javac | 5m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 19s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 18s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/23/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 40s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/23/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed |
| +1 :green_heart: | spotbugs | 14m 2s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 30s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 5m 59s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 27s | | tez-examples in the patch
passed. |
| +1 :green_heart: | unit | 5m 17s | | tez-dag in the patch passed. |
| +1 :green_heart: | unit | 71m 42s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 13s | | The patch does not
generate ASF License warnings. |
| | | 164m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/23/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/tez/pull/427 |
| Optional Tests | dupname asflicense javac javadoc unit codespell
detsecrets xmllint compile spotbugs checkstyle |
| uname | Linux 30c717337d59 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11
08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
tez-yetus commented on PR #427:
URL: https://github.com/apache/tez/pull/427#issuecomment-3555045182
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 23m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 2m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 10m 37s | | master passed |
| +1 :green_heart: | compile | 5m 1s | | master passed |
| +1 :green_heart: | checkstyle | 2m 50s | | master passed |
| +1 :green_heart: | javadoc | 3m 35s | | master passed |
| +0 :ok: | spotbugs | 2m 10s | | tez-api in master has 610 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 43s | | tez-common in master has 13 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 8s | | tez-runtime-library in master has
241 extant spotbugs warnings. |
| +0 :ok: | spotbugs | 0m 41s | | tez-examples in master has 2 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 1m 41s | | tez-dag in master has 785 extant
spotbugs warnings. |
| +0 :ok: | spotbugs | 6m 57s | | root in master has 2066 extant
spotbugs warnings. |
| -0 :warning: | patch | 7m 40s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 35s | | the patch passed |
| +1 :green_heart: | codespell | 0m 52s | | No new issues. |
| +1 :green_heart: | compile | 5m 2s | | the patch passed |
| +1 :green_heart: | javac | 5m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | tez-api: The patch
generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121) |
| +1 :green_heart: | checkstyle | 0m 18s | | The patch passed
checkstyle in tez-common |
| +1 :green_heart: | checkstyle | 0m 23s | | The patch passed
checkstyle in tez-runtime-library |
| +1 :green_heart: | checkstyle | 0m 19s | | tez-examples: The patch
generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/results-checkstyle-tez-dag.txt)
| tez-dag: The patch generated 1 new + 610 unchanged - 2 fixed = 611 total
(was 612) |
| -0 :warning: | checkstyle | 0m 41s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 740 unchanged - 6 fixed = 741 total (was
746) |
| +1 :green_heart: | javadoc | 3m 30s | | the patch passed |
| -1 :x: | spotbugs | 1m 58s |
[/new-spotbugs-tez-dag.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/new-spotbugs-tez-dag.html)
| tez-dag generated 1 new + 785 unchanged - 0 fixed = 786 total (was 785) |
| -1 :x: | spotbugs | 7m 17s |
[/new-spotbugs-root.html](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/new-spotbugs-root.html)
| root generated 1 new + 2066 unchanged - 0 fixed = 2067 total (was 2066) |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 32s | | tez-api in the patch passed. |
| +1 :green_heart: | unit | 0m 39s | | tez-common in the patch passed.
|
| +1 :green_heart: | unit | 5m 56s | | tez-runtime-library in the
patch passed. |
| +1 :green_heart: | unit | 0m 27s | | tez-examples in the patch
passed. |
| -1 :x: | unit | 5m 17s |
[/patch-unit-tez-dag.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/patch-unit-tez-dag.txt)
| tez-dag in the patch passed. |
| -1 :x: | unit | 69m 48s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/21/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 11s | | The patch does not
generate ASF License warnings. |
| | | 186m 13s | | |
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2543404391
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
+ private ZkAMRegistryClient amRegistryClient = null;
+ private volatile boolean isRunning = false;
+ private String amHost;
+ private int amPort;
+
+ @Override
+ public synchronized void init(TezConfiguration tezConf) {
+if (this.amRegistryClient == null) {
+ try {
+this.amRegistryClient = ZkAMRegistryClient.getClient(tezConf);
+this.isRunning = true;
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ this.amRegistryClient.start();
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+isRunning = true;
+ }
+
+ @Override
+ public void stop() {
+isRunning = false;
+amRegistryClient.close();
+ }
+
+ @Override
+ public void close() throws IOException {
+amRegistryClient.close();
+ }
+
+ @Override
+ public YarnClientApplication createApplication() throws YarnException,
IOException {
+ApplicationSubmissionContext context =
Records.newRecord(ApplicationSubmissionContext.class);
+ApplicationId appId = amRecord.getApplicationId();
Review Comment:
right, confusing, added javadoc comment for clarity:
```
* Hidden assumption here: this method assumes that
* {@code getApplicationReport()} has already been called before
* {@code createApplication()}, ensuring that {@code
amRecord.getApplicationId()}
* is always available. This assumption holds in all supported usage
patterns:
* the only code path where {@code createApplication()} might be called
first is
* {@code TezClient.submitDAGApplication()}, but that path is never
exercised in
* Zookeeper standalone mode because that mode assumes applications are
already
* running. Therefore, the ordering guarantee is valid in practice.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2543212110
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/yarn/YarnServerFrameworkService.java:
##
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.yarn;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.yarn.api.ApplicationConstants;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.util.ConverterUtils;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.common.TezUtilsInternal;
+import org.apache.tez.common.security.JobTokenIdentifier;
+import org.apache.tez.common.security.JobTokenSecretManager;
+import org.apache.tez.common.security.TokenCache;
+import org.apache.tez.dag.api.TezConstants;
+import org.apache.tez.dag.api.records.DAGProtos;
+import org.apache.tez.dag.app.ClusterInfo;
+import org.apache.tez.dag.app.dag.Vertex;
+import org.apache.tez.frameworkplugins.AmExtensions;
+import org.apache.tez.frameworkplugins.ServerFrameworkService;
+import org.apache.tez.frameworkplugins.TaskResourceException;
+
+/**
+ * YARN-based server framework service implementation.
+ * Provides default YARN framework server functionality with default
implementations
+ * for all AmExtensions methods.
+ */
+public class YarnServerFrameworkService implements ServerFrameworkService {
+
+ private final YarnAmExtensions amExtensions = new YarnAmExtensions();
+
+ @Override
+ public Optional createOrGetAMRegistry(Configuration conf) {
+// Return empty for now - YARN mode doesn't require a custom AM registry
+return Optional.empty();
+ }
+
+ @Override
+ public AmExtensions createAmExtensions() {
+return new YarnAmExtensions();
Review Comment:
good catch, let me fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
okumin commented on code in PR #427:
URL: https://github.com/apache/tez/pull/427#discussion_r2520588552
##
tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java:
##
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tez.client.registry.zookeeper;
+
+import java.io.IOException;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClientApplication;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Records;
+import org.apache.tez.client.FrameworkClient;
+import org.apache.tez.client.registry.AMRecord;
+import org.apache.tez.dag.api.TezConfiguration;
+
+public class ZkFrameworkClient extends FrameworkClient {
+
+ private AMRecord amRecord;
+ private ZkAMRegistryClient amRegistryClient = null;
+ private volatile boolean isRunning = false;
+ private String amHost;
+ private int amPort;
+
+ @Override
+ public synchronized void init(TezConfiguration tezConf) {
+if (this.amRegistryClient == null) {
+ try {
+this.amRegistryClient = ZkAMRegistryClient.getClient(tezConf);
+this.isRunning = true;
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+}
+ }
+
+ @Override
+ public void start() {
+try {
+ this.amRegistryClient.start();
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+isRunning = true;
+ }
+
+ @Override
+ public void stop() {
+isRunning = false;
+amRegistryClient.close();
+ }
+
+ @Override
+ public void close() throws IOException {
+amRegistryClient.close();
+ }
+
+ @Override
+ public YarnClientApplication createApplication() throws YarnException,
IOException {
+ApplicationSubmissionContext context =
Records.newRecord(ApplicationSubmissionContext.class);
+ApplicationId appId = amRecord.getApplicationId();
Review Comment:
I'm not understanding how Tez can bootstrap `amRecord`, which requires
`getApplicationReport` to be invoked with an application id. Probably, I'm
overlooking something
##
tez-dag/src/main/java/org/apache/tez/frameworkplugins/yarn/YarnServerFrameworkService.java:
##
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tez.frameworkplugins.yarn;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.yarn.api.ApplicationConstants;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.util.ConverterUtils;
+import org.apache.tez.client.registry.AMRegistry;
+import org.apache.tez.common.TezUtilsInternal;
+import org.apache.tez.common.security.JobTokenIdentifier;
+import or
Re: [PR] TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices [tez]
abstractdog commented on PR #427: URL: https://github.com/apache/tez/pull/427#issuecomment-347955 @ayushtkn, @okumin, @zhangbutao: this is ready for reviews now please be aware that all the reported checkstyle/codespell/asflicense problems are not present any, becase they belong to classes that I removed, I created [TEZ-4660](https://issues.apache.org/jira/browse/TEZ-4660) about this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
