[jira] [Commented] (HADOOP-18602) Remove netty3 dependency
[ https://issues.apache.org/jira/browse/HADOOP-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683253#comment-17683253 ] ASF GitHub Bot commented on HADOOP-18602: - tomicooler commented on PR #5323: URL: https://github.com/apache/hadoop/pull/5323#issuecomment-1413290947 Hi @steveloughran, [HADOOP-15327](https://issues.apache.org/jira/browse/HADOOP-15327) was not backported to branch-3.3 so mapreduce still uses netty3 in that branch. > Remove netty3 dependency > > > Key: HADOOP-18602 > URL: https://issues.apache.org/jira/browse/HADOOP-18602 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 3.4.0 >Reporter: Tamas Domok >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > AFAIK netty3 is no longer in use so it can be removed from the dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on pull request #5323: HADOOP-18602. Remove netty3 dependency
tomicooler commented on PR #5323: URL: https://github.com/apache/hadoop/pull/5323#issuecomment-1413290947 Hi @steveloughran, [HADOOP-15327](https://issues.apache.org/jira/browse/HADOOP-15327) was not backported to branch-3.3 so mapreduce still uses netty3 in that branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hfutatzhanghb opened a new pull request, #5338: [HDFS-16903]. Fix javadoc of LightWeightResizableGSet class
hfutatzhanghb opened a new pull request, #5338: URL: https://github.com/apache/hadoop/pull/5338 After [HDFS-16249. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl.], the Class LightWeightResizableGSet is thread-safe. So we should fix the docs of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
[ https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen reassigned HADOOP-18610: - Assignee: (was: Haifeng Chen) > ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS > - > > Key: HADOOP-18610 > URL: https://issues.apache.org/jira/browse/HADOOP-18610 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 3.3.4 >Reporter: Haifeng Chen >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity > with with [Azure Active Directory (Azure AD) workload > identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] > (preview), which integrate with the Kubernetes native capabilities to > federate with any external identity providers. This approach is simpler to > use and deploy. > Refer to > [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview|https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] > and [https://azure.github.io/azure-workload-identity/docs/introduction.html] > for more details. > The basic use scenario is to access Azure cloud resources (such as cloud > storage) from Kubernetes (such as AKS) workload using Azure managed identity > federated with Kubernetes service account. The credential environment > variables in pod projected by Azure AD workload identity are like following: > AZURE_AUTHORITY_HOST: (Injected by the webhook, > [https://login.microsoftonline.com/]) > AZURE_CLIENT_ID: (Injected by the webhook) > AZURE_TENANT_ID: (Injected by the webhook) > AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, > /var/run/secrets/azure/tokens/azure-identity-token) > The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON > Web Token) client assertion token which we can use to request to > AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId + > "/oauth2/v2.0/token") for a AD token which can be used to directly access > the Azure cloud resources. > This approach is very common and similar among cloud providers such as AWS > and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to > handle the same case. > The existing MsiTokenProvider can only handle the managed identity associated > with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider > which handle Azure Workload Identity case. For this, we need to add one > method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used > by WorkloadIdentityTokenProvider. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice
tomscut commented on code in PR #5334: URL: https://github.com/apache/hadoop/pull/5334#discussion_r1094080750 ## hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html: ## @@ -81,6 +81,7 @@ Namenode Address + Namenode HA state Review Comment: nit: `state` -> `State`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683235#comment-17683235
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1094072017
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
@steveloughran @jojochuang @Apache9 WDYT we should do here?
On one hand, we want to remove commons-logging loggers from code base,
that's the main purpose of this PR.
On the other hand, removing commons-logging means we remove this method
because it's not in use and it has direct reference to commons-logging logger,
and the method is IA.Public.
If we deprecate this method and keep it, we are not done with the purpose of
this PR i.e. cleaning up of commons-logging.
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1094072017
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
@steveloughran @jojochuang @Apache9 WDYT we should do here?
On one hand, we want to remove commons-logging loggers from code base,
that's the main purpose of this PR.
On the other hand, removing commons-logging means we remove this method
because it's not in use and it has direct reference to commons-logging logger,
and the method is IA.Public.
If we deprecate this method and keep it, we are not done with the purpose of
this PR i.e. cleaning up of commons-logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath
hadoop-yetus commented on PR #5276:
URL: https://github.com/apache/hadoop/pull/5276#issuecomment-1413196341
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5276/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 2
unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 39m 57s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 170m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5276/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5276 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux efacb027686e 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1
17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d9318ef7a50e8ef48425a9e0aa0d6b4aae4c4309 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5276/5/testReport/ |
| Max. process+thread count | 2454 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5276/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message fro
[GitHub] [hadoop] snmvaughan opened a new pull request, #5337: MAPREDUCE-7386. Maven parallel builds (skipping tests) fail
snmvaughan opened a new pull request, #5337: URL: https://github.com/apache/hadoop/pull/5337 ### Description of PR Backport from trunk. This matches the changes made to trunk that allow for parallel building of a distribution, and has been in use since November of last year. ### How was this patch tested? The patch was tested using the Hadoop development environment docker image. In addition to verifying the distribution, I've compared the published dependencies to verify that the changes are limited to the modified modules. The changes allow Maven to determine a correct build graph for safely performing parallel builds. ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
[ https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HADOOP-18610: -- Description: In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity with with [Azure Active Directory (Azure AD) workload identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] (preview), which integrate with the Kubernetes native capabilities to federate with any external identity providers. This approach is simpler to use and deploy. Refer to [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview|https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] and [https://azure.github.io/azure-workload-identity/docs/introduction.html] for more details. The basic use scenario is to access Azure cloud resources (such as cloud storage) from Kubernetes (such as AKS) workload using Azure managed identity federated with Kubernetes service account. The credential environment variables in pod projected by Azure AD workload identity are like following: AZURE_AUTHORITY_HOST: (Injected by the webhook, [https://login.microsoftonline.com/]) AZURE_CLIENT_ID: (Injected by the webhook) AZURE_TENANT_ID: (Injected by the webhook) AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, /var/run/secrets/azure/tokens/azure-identity-token) The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON Web Token) client assertion token which we can use to request to AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId + "/oauth2/v2.0/token") for a AD token which can be used to directly access the Azure cloud resources. This approach is very common and similar among cloud providers such as AWS and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to handle the same case. The existing MsiTokenProvider can only handle the managed identity associated with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider which handle Azure Workload Identity case. For this, we need to add one method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used by WorkloadIdentityTokenProvider. was: In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity with with [Azure Active Directory (Azure AD) workload identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] (preview), which integrate with the Kubernetes native capabilities to federate with any external identity providers. This approach is simpler to use and deploy. Refer to for more details: [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] The basic use scenario is to access Azure cloud resources (such as cloud storage) from Kubernetes (such as AKS) workload using Azure managed identity federated with Kubernetes service account. The credential environment variables in pod projected by Azure AD workload identity are like following: AZURE_AUTHORITY_HOST: (Injected by the webhook, https://login.microsoftonline.com/) AZURE_CLIENT_ID: (Injected by the webhook) AZURE_TENANT_ID: (Injected by the webhook) AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, /var/run/secrets/azure/tokens/azure-identity-token) The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON Web Token) client assertion token which we can use to request to AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId + "/oauth2/v2.0/token") for a AD token which can be used to directly access the Azure cloud resources. This approach is very common and similar among cloud providers such as AWS and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to handle the same case. The existing MsiTokenProvider can only handle the managed identity associated with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider which handle Azure Workload Identity case. For this, we need to add one method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used by WorkloadIdentityTokenProvider. > ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS > - > > Key: HADOOP-18610 > URL: https://issues.apache.org/jira/browse/HADOOP-18610 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 3.3.4 >Reporter: Haifeng Chen >Assignee: Haifeng Chen >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity > with with [Azure Active Directory (Azure AD) workload > identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-id
[jira] [Created] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
Haifeng Chen created HADOOP-18610: - Summary: ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS Key: HADOOP-18610 URL: https://issues.apache.org/jira/browse/HADOOP-18610 Project: Hadoop Common Issue Type: Improvement Components: tools Affects Versions: 3.3.4 Reporter: Haifeng Chen In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity with with [Azure Active Directory (Azure AD) workload identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] (preview), which integrate with the Kubernetes native capabilities to federate with any external identity providers. This approach is simpler to use and deploy. Refer to for more details: [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] The basic use scenario is to access Azure cloud resources (such as cloud storage) from Kubernetes (such as AKS) workload using Azure managed identity federated with Kubernetes service account. The credential environment variables in pod projected by Azure AD workload identity are like following: AZURE_AUTHORITY_HOST: (Injected by the webhook, https://login.microsoftonline.com/) AZURE_CLIENT_ID: (Injected by the webhook) AZURE_TENANT_ID: (Injected by the webhook) AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, /var/run/secrets/azure/tokens/azure-identity-token) The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON Web Token) client assertion token which we can use to request to AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId + "/oauth2/v2.0/token") for a AD token which can be used to directly access the Azure cloud resources. This approach is very common and similar among cloud providers such as AWS and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to handle the same case. The existing MsiTokenProvider can only handle the managed identity associated with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider which handle Azure Workload Identity case. For this, we need to add one method (getTokenUsingJWTAssertion) in AzureADAuthenticator which will be used by WorkloadIdentityTokenProvider. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18610) ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS
[ https://issues.apache.org/jira/browse/HADOOP-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen reassigned HADOOP-18610: - Assignee: Haifeng Chen > ABFS OAuth2 Token Provider to support Azure Workload Identity for AKS > - > > Key: HADOOP-18610 > URL: https://issues.apache.org/jira/browse/HADOOP-18610 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 3.3.4 >Reporter: Haifeng Chen >Assignee: Haifeng Chen >Priority: Critical > Original Estimate: 168h > Remaining Estimate: 168h > > In Jan 2023, Microsoft Azure AKS replaced its original pod-managed identity > with with [Azure Active Directory (Azure AD) workload > identities|https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identities-overview] > (preview), which integrate with the Kubernetes native capabilities to > federate with any external identity providers. This approach is simpler to > use and deploy. > Refer to for more details: > [https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview.] > The basic use scenario is to access Azure cloud resources (such as cloud > storage) from Kubernetes (such as AKS) workload using Azure managed identity > federated with Kubernetes service account. The credential environment > variables in pod projected by Azure AD workload identity are like following: > AZURE_AUTHORITY_HOST: (Injected by the webhook, > https://login.microsoftonline.com/) > AZURE_CLIENT_ID: (Injected by the webhook) > AZURE_TENANT_ID: (Injected by the webhook) > AZURE_FEDERATED_TOKEN_FILE: (Injected by the webhook, > /var/run/secrets/azure/tokens/azure-identity-token) > > The token in the file pointed by AZURE_FEDERATED_TOKEN_FILE is a JWT (JASON > Web Token) client assertion token which we can use to request to > AZURE_AUTHORITY_HOST (url is AZURE_AUTHORITY_HOST + tenantId + > "/oauth2/v2.0/token") for a AD token which can be used to directly access > the Azure cloud resources. > This approach is very common and similar among cloud providers such as AWS > and GCP. Hadoop AWS integration has WebIdentityTokenCredentialProvider to > handle the same case. > The existing MsiTokenProvider can only handle the managed identity associated > with Azure VM instance. We need to implement a WorkloadIdentityTokenProvider > which handle Azure Workload Identity case. For this, we need to add one > method (getTokenUsingJWTAssertion) > in AzureADAuthenticator which will be used by WorkloadIdentityTokenProvider. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hfutatzhanghb commented on a diff in pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath
hfutatzhanghb commented on code in PR #5276:
URL: https://github.com/apache/hadoop/pull/5276#discussion_r1093968053
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java:
##
@@ -441,9 +448,12 @@ public PathLocation getDestinationForPath(final String
path)
if (this.locationCache == null) {
res = lookupLocation(processTrashPath(path));
} else {
-Callable meh = (Callable) () ->
-lookupLocation(processTrashPath(path));
+Callable meh = (Callable) () -> {
+ LOC_CACHE_MISS.increment();
+ return lookupLocation(processTrashPath(path));
+};
res = this.locationCache.get(processTrashPath(path), meh);
+LOC_CACHE_ACCESS.increment();
Review Comment:
hi, @goiri @tomscut , sorry for responding so late. i have add an unit test
to test the cache access. could you please help me to review the code? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hfutatzhanghb commented on a diff in pull request #5276: HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath
hfutatzhanghb commented on code in PR #5276:
URL: https://github.com/apache/hadoop/pull/5276#discussion_r1093968053
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java:
##
@@ -441,9 +448,12 @@ public PathLocation getDestinationForPath(final String
path)
if (this.locationCache == null) {
res = lookupLocation(processTrashPath(path));
} else {
-Callable meh = (Callable) () ->
-lookupLocation(processTrashPath(path));
+Callable meh = (Callable) () -> {
+ LOC_CACHE_MISS.increment();
+ return lookupLocation(processTrashPath(path));
+};
res = this.locationCache.get(processTrashPath(path), meh);
+LOC_CACHE_ACCESS.increment();
Review Comment:
hi, @goiri @tomscut , sorry for responding so late. i have add an unit test
to test the cache access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hellozepp closed pull request #5336: Add common ugi cnf
hellozepp closed pull request #5336: Add common ugi cnf URL: https://github.com/apache/hadoop/pull/5336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hellozepp opened a new pull request, #5336: Add common ugi cnf
hellozepp opened a new pull request, #5336: URL: https://github.com/apache/hadoop/pull/5336 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice
virajjasani commented on code in PR #5334:
URL: https://github.com/apache/hadoop/pull/5334#discussion_r1093941288
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -202,6 +202,7 @@ private String getNameNodeAddress() {
Map getActorInfoMap() {
final Map info = new HashMap();
info.put("NamenodeAddress", getNameNodeAddress());
+info.put("NamenodeState", state.toString());
Review Comment:
Actually that is better, since that is the main purpose we want to debug.
Let me make this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
slfan1989 commented on code in PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#discussion_r1093934555
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java:
##
@@ -464,7 +464,7 @@ public int compareTo(Resource other) {
@Override
public String toString() {
-return getFormattedString(String.valueOf(getMemorySize()));
+return getFormattedString();
Review Comment:
Thanks for your suggestion, I will modify the code of the NodeLabel page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
slfan1989 commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1093933058
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime() throws YarnException,
InterruptedException {
+Thread.sleep(1000);
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
slfan1989 commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1093932317
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/SubClusterState.java:
##
@@ -55,6 +55,10 @@ public boolean isUnusable() {
return (this != SC_RUNNING && this != SC_NEW);
}
+ public boolean isUsable() {
+return !isUnusable();
Review Comment:
Thank you very much for helping to review the code, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a diff in pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice
liuml07 commented on code in PR #5334:
URL: https://github.com/apache/hadoop/pull/5334#discussion_r1093890563
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -202,6 +202,7 @@ private String getNameNodeAddress() {
Map getActorInfoMap() {
final Map info = new HashMap();
info.put("NamenodeAddress", getNameNodeAddress());
+info.put("NamenodeState", state.toString());
Review Comment:
nit: is it better to name it `NameNodeHaState`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
goiri commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1093763964
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/SubClusterState.java:
##
@@ -55,6 +55,10 @@ public boolean isUnusable() {
return (this != SC_RUNNING && this != SC_NEW);
}
+ public boolean isUsable() {
+return !isUnusable();
Review Comment:
Can we just use this directly?
Let's just have one of them either isUsable or isUnusable.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime() throws YarnException,
InterruptedException {
+Thread.sleep(1000);
Review Comment:
Could we wait for that attribute in all clusters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683153#comment-17683153 ] ASF GitHub Bot commented on HADOOP-18279: - hadoop-yetus commented on PR #4789: URL: https://github.com/apache/hadoop/pull/4789#issuecomment-1412762405 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 21s | | https://github.com/apache/hadoop/pull/4789 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/4789 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4789/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4789: HADOOP-18279. Cancel fileMonitoringTimer even if trustManager isn't defined
hadoop-yetus commented on PR #4789: URL: https://github.com/apache/hadoop/pull/4789#issuecomment-1412762405 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 21s | | https://github.com/apache/hadoop/pull/4789 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/4789 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4789/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18279: -- Fix Version/s: 3.3.9 > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683152#comment-17683152 ] ASF GitHub Bot commented on HADOOP-18279: - sunchao commented on PR #4789: URL: https://github.com/apache/hadoop/pull/4789#issuecomment-1412760608 Thanks, merged to `branch-3.3` > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683151#comment-17683151 ] ASF GitHub Bot commented on HADOOP-18279: - sunchao merged PR #4789: URL: https://github.com/apache/hadoop/pull/4789 > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #4789: HADOOP-18279. Cancel fileMonitoringTimer even if trustManager isn't defined
sunchao commented on PR #4789: URL: https://github.com/apache/hadoop/pull/4789#issuecomment-1412760608 Thanks, merged to `branch-3.3` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao merged pull request #4789: HADOOP-18279. Cancel fileMonitoringTimer even if trustManager isn't defined
sunchao merged PR #4789: URL: https://github.com/apache/hadoop/pull/4789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-18279. --- Resolution: Fixed > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683132#comment-17683132
]
ASF GitHub Bot commented on HADOOP-18592:
-
cnauroth commented on PR #5294:
URL: https://github.com/apache/hadoop/pull/5294#issuecomment-1412561222
@liuml07 , thanks for committing. I got delayed. @virajjasani , thank you
for the patch!
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on pull request #5294: HADOOP-18592 Sasl connection failure should log remote address
cnauroth commented on PR #5294: URL: https://github.com/apache/hadoop/pull/5294#issuecomment-1412561222 @liuml07 , thanks for committing. I got delayed. @virajjasani , thank you for the patch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
goiri commented on code in PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#discussion_r1093596818
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java:
##
@@ -464,7 +464,7 @@ public int compareTo(Resource other) {
@Override
public String toString() {
-return getFormattedString(String.valueOf(getMemorySize()));
+return getFormattedString();
Review Comment:
Isn't it dangerous to do it for everything and not just for the UI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-18592.
Fix Version/s: 3.4.0
3.3.9
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to branch-3.3 and trunk branches. Thank you for your contribution,
[~vjasani] . Thank you all for review.
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5162: HDFS-16853. BugFix HADOOP-18324 caused UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
steveloughran commented on code in PR #5162:
URL: https://github.com/apache/hadoop/pull/5162#discussion_r1093579485
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:
##
@@ -1182,7 +1183,12 @@ public void sendRpcRequest(final Call call)
final ResponseBuffer buf = new ResponseBuffer();
header.writeDelimitedTo(buf);
RpcWritable.wrap(call.rpcRequest).writeTo(buf);
- rpcRequestQueue.put(Pair.of(call, buf));
+ synchronized (sendRpcRequestLock) {
Review Comment:
markClosed doesn't use the same lock, so i don't see this stopping a race
condition between
this submission and the markClosed set+purge queue;
better: move rpcRequestQueue.put(Pair.of(call, buf)); into a new method, and
make synchronized
```java
private sychronized put(pair) {
if (!shouldCloseConnection.get()) {
rpcRequestQueue.put(pair);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683122#comment-17683122
]
ASF GitHub Bot commented on HADOOP-18592:
-
liuml07 merged PR #5294:
URL: https://github.com/apache/hadoop/pull/5294
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 merged pull request #5294: HADOOP-18592 Sasl connection failure should log remote address
liuml07 merged PR #5294: URL: https://github.com/apache/hadoop/pull/5294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18602) Remove netty3 dependency
[ https://issues.apache.org/jira/browse/HADOOP-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683121#comment-17683121 ] ASF GitHub Bot commented on HADOOP-18602: - steveloughran commented on PR #5323: URL: https://github.com/apache/hadoop/pull/5323#issuecomment-1412506676 what about branch 3.3? > Remove netty3 dependency > > > Key: HADOOP-18602 > URL: https://issues.apache.org/jira/browse/HADOOP-18602 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 3.4.0 >Reporter: Tamas Domok >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > AFAIK netty3 is no longer in use so it can be removed from the dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5323: HADOOP-18602. Remove netty3 dependency
steveloughran commented on PR #5323: URL: https://github.com/apache/hadoop/pull/5323#issuecomment-1412506676 what about branch 3.3? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5301: HDFS-16892. Fix method name of RPC.Builder#setnumReaders
steveloughran commented on code in PR #5301:
URL: https://github.com/apache/hadoop/pull/5301#discussion_r1093569537
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java:
##
@@ -901,7 +901,7 @@ public Builder setNumHandlers(int numHandlers) {
* @return Default: -1.
* @param numReaders input numReaders.
*/
-public Builder setnumReaders(int numReaders) {
+public Builder setNumReaders(int numReaders) {
Review Comment:
it'll break things. at best a new method can be added, internal code moved
and this one tagged as deprecated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[
https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683117#comment-17683117
]
ASF GitHub Bot commented on HADOOP-18606:
-
steveloughran commented on code in PR #5299:
URL: https://github.com/apache/hadoop/pull/5299#discussion_r1093487300
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java:
##
@@ -111,6 +111,15 @@ public final class AbfsHttpConstants {
public static final char CHAR_EQUALS = '=';
public static final char CHAR_STAR = '*';
public static final char CHAR_PLUS = '+';
+ /**
Review Comment:
javadoc nits.
1. add a line break.
2. for those lines you want well laid out, just use a \ above and a
below, remove those \ tags
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -237,6 +240,11 @@ private void completeExecute(TracingContext tracingContext)
LOG.trace("{} REST operation complete", operationType);
}
+ @VisibleForTesting
+ String getClientLatency() {
+return this.client.getAbfsPerfTracker().getClientLatency();
Review Comment:
remove the `this.`. I know, it was there before, but this is time to clean
up.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -73,6 +74,8 @@ public class AbfsRestOperation {
private AbfsHttpOperation result;
private AbfsCounters abfsCounters;
+ private String failureReason = null;
Review Comment:
add a javadoc to explain. no need to set as null as that is the default and
it actually speeds up object creation to not set it
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -334,6 +347,11 @@ private boolean executeHttpOperation(final int retryCount,
return true;
}
+ @VisibleForTesting
+ AbfsHttpOperation getHttpOperation() throws IOException {
Review Comment:
as this is more than just a getter, can you
* call it `createHttpOperation()`
* add a javadoc description
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RetryReason.java:
##
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.net.SocketException;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HTTP_STATUS_CATEGORY_QUOTIENT;
+
+/**
+ * In case of retry, this enum would give the information on the reason for
+ * previous API call.
+ * */
+public enum RetryReason {
+ CONNECTION_TIMEOUT(2,
+ ((exceptionCaptured, statusCode, serverErrorMessage) -> {
+if (exceptionCaptured != null && "connect timed out".equalsIgnoreCase(
Review Comment:
can you do a line break before the &&
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RetryReason.java:
##
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOExc
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5299: HADOOP-18606. Add reason in in x-ms-client-request-id on a retry API call.
steveloughran commented on code in PR #5299:
URL: https://github.com/apache/hadoop/pull/5299#discussion_r1093487300
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java:
##
@@ -111,6 +111,15 @@ public final class AbfsHttpConstants {
public static final char CHAR_EQUALS = '=';
public static final char CHAR_STAR = '*';
public static final char CHAR_PLUS = '+';
+ /**
Review Comment:
javadoc nits.
1. add a line break.
2. for those lines you want well laid out, just use a \ above and a
below, remove those \ tags
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -237,6 +240,11 @@ private void completeExecute(TracingContext tracingContext)
LOG.trace("{} REST operation complete", operationType);
}
+ @VisibleForTesting
+ String getClientLatency() {
+return this.client.getAbfsPerfTracker().getClientLatency();
Review Comment:
remove the `this.`. I know, it was there before, but this is time to clean
up.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -73,6 +74,8 @@ public class AbfsRestOperation {
private AbfsHttpOperation result;
private AbfsCounters abfsCounters;
+ private String failureReason = null;
Review Comment:
add a javadoc to explain. no need to set as null as that is the default and
it actually speeds up object creation to not set it
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -334,6 +347,11 @@ private boolean executeHttpOperation(final int retryCount,
return true;
}
+ @VisibleForTesting
+ AbfsHttpOperation getHttpOperation() throws IOException {
Review Comment:
as this is more than just a getter, can you
* call it `createHttpOperation()`
* add a javadoc description
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RetryReason.java:
##
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.net.SocketException;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HTTP_STATUS_CATEGORY_QUOTIENT;
+
+/**
+ * In case of retry, this enum would give the information on the reason for
+ * previous API call.
+ * */
+public enum RetryReason {
+ CONNECTION_TIMEOUT(2,
+ ((exceptionCaptured, statusCode, serverErrorMessage) -> {
+if (exceptionCaptured != null && "connect timed out".equalsIgnoreCase(
Review Comment:
can you do a line break before the &&
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RetryReason.java:
##
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.net.SocketException;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpC
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683116#comment-17683116
]
ASF GitHub Bot commented on HADOOP-18206:
-
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412484123
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 30s | | trunk passed |
| +1 :green_heart: | compile | 31m 46s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 26m 40s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 29m 6s | | trunk passed |
| -1 :x: | javadoc | 1m 29s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 5m 1s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 41m 40s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 67m 8s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 67m 26s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 47m 55s | | the patch passed |
| +1 :green_heart: | compile | 30m 41s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 30m 41s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2814 unchanged - 5
fixed = 2814 total (was 2819) |
| +1 :green_heart: | compile | 26m 37s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 26m 37s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2612 unchanged - 5 fixed
= 2612 total (was 2617) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 56s | | root: The patch generated
0 new + 683 unchanged - 28 fixed = 683 total (was 711) |
| +1 :green_heart: | mvnsite | 24m 14s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| -1 :x: | javadoc | 1m 21s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed wit
[GitHub] [hadoop] hadoop-yetus commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412484123
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 30s | | trunk passed |
| +1 :green_heart: | compile | 31m 46s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 26m 40s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 29m 6s | | trunk passed |
| -1 :x: | javadoc | 1m 29s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 5m 1s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 41m 40s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 67m 8s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 67m 26s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 47m 55s | | the patch passed |
| +1 :green_heart: | compile | 30m 41s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 30m 41s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2814 unchanged - 5
fixed = 2814 total (was 2819) |
| +1 :green_heart: | compile | 26m 37s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 26m 37s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2612 unchanged - 5 fixed
= 2612 total (was 2617) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 56s | | root: The patch generated
0 new + 683 unchanged - 28 fixed = 683 total (was 711) |
| +1 :green_heart: | mvnsite | 24m 14s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| -1 :x: | javadoc | 1m 21s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/22/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 67m 4s | | patch has no errors
when building and testing our client arti
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683107#comment-17683107
]
ASF GitHub Bot commented on HADOOP-18206:
-
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412447231
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 11s | | trunk passed |
| +1 :green_heart: | compile | 31m 43s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 26m 30s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 29m 6s | | trunk passed |
| -1 :x: | javadoc | 1m 28s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 12s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 17s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 5m 9s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 41m 29s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 67m 2s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 67m 20s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 48m 6s | | the patch passed |
| +1 :green_heart: | compile | 30m 56s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 30m 56s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 26m 43s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 26m 43s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2612 unchanged - 5 fixed
= 2612 total (was 2617) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 1s | | root: The patch generated
0 new + 683 unchanged - 28 fixed = 683 total (was 711) |
| +1 :green_heart: | mvnsite | 23m 48s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 17s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 25s | | the patch passed wit
[GitHub] [hadoop] hadoop-yetus commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412447231
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 11s | | trunk passed |
| +1 :green_heart: | compile | 31m 43s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 26m 30s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 29m 6s | | trunk passed |
| -1 :x: | javadoc | 1m 28s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 12s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 17s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 5m 9s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 41m 29s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 67m 2s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 67m 20s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 48m 6s | | the patch passed |
| +1 :green_heart: | compile | 30m 56s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 30m 56s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 26m 43s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 26m 43s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2612 unchanged - 5 fixed
= 2612 total (was 2617) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 1s | | root: The patch generated
0 new + 683 unchanged - 28 fixed = 683 total (was 711) |
| +1 :green_heart: | mvnsite | 23m 48s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 17s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/21/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 25s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 17s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 67m 29s | | patch has no errors
when building and testing our client arti
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683106#comment-17683106
]
ASF GitHub Bot commented on HADOOP-18206:
-
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412433010
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 21s | | trunk passed |
| +1 :green_heart: | compile | 23m 8s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 40s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 27m 7s | | trunk passed |
| -1 :x: | javadoc | 1m 26s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 16s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 3m 50s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 31m 57s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 54m 26s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 54m 47s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 42m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 37s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 37s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 33s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2613 unchanged - 5 fixed
= 2613 total (was 2618) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 42s | | root: The patch generated
0 new + 684 unchanged - 28 fixed = 684 total (was 712) |
| +1 :green_heart: | mvnsite | 19m 51s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 13s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 17s | | the patch passed wit
[GitHub] [hadoop] hadoop-yetus commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1412433010
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 21s | | trunk passed |
| +1 :green_heart: | compile | 23m 8s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 40s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 27m 7s | | trunk passed |
| -1 :x: | javadoc | 1m 26s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 16s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 3m 50s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 31m 57s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 54m 26s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 54m 47s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 42m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 37s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 37s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 33s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2613 unchanged - 5 fixed
= 2613 total (was 2618) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 42s | | root: The patch generated
0 new + 684 unchanged - 28 fixed = 684 total (was 712) |
| +1 :green_heart: | mvnsite | 19m 51s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 1m 13s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/20/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 21s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 54m 43s | | patch has no errors
when building and testing our client arti
[jira] [Commented] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683094#comment-17683094 ] ASF GitHub Bot commented on HADOOP-18606: - steveloughran commented on PR #5299: URL: https://github.com/apache/hadoop/pull/5299#issuecomment-1412380738 will review...been offline for most of january > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5299: HADOOP-18606. Add reason in in x-ms-client-request-id on a retry API call.
steveloughran commented on PR #5299: URL: https://github.com/apache/hadoop/pull/5299#issuecomment-1412380738 will review...been offline for most of january -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
brumi1024 commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1093432627
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -723,6 +727,17 @@ public OrderingPolicy
getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+setBoolean(getQueuePrefix(queue) +
+"ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy,
String postfix) {
+return getBoolean(getQueuePrefix(queue) +
+"ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
This naming is really misleading, as it doesn't get the ordering policy, but
it gets the properties of that policy. Since the ordering policy is pluggable,
anyone can write one I think we shouldn't introduce getters for it, especially
not fixed return type ones.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -536,6 +536,10 @@ public void
setMaximumApplicationMasterResourcePerQueuePercent(String queue,
setFloat(getQueuePrefix(queue) + MAXIMUM_AM_RESOURCE_SUFFIX, percent);
}
+ public void setMaximumApplicationMasterResourcePercent(float percent) {
Review Comment:
Nit: a getter exists for this property, could you please move and rename
this setter to be more in line of that method? It's called:
getMaximumApplicationMasterResourcePercent
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -1203,6 +1218,16 @@ public void reinitializeConfigurationProperties() {
configurationProperties = new ConfigurationProperties(props);
}
+ public void setQueueMaximumAllocationMb(String queue, String value) {
+String queuePrefix = getQueuePrefix(queue);
+set(queuePrefix + "maximum-allocation-mb", value);
Review Comment:
Please use the static constants instead of literals.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -1693,6 +1726,14 @@ public boolean getIntraQueuePreemptionDisabled(String
queue,
+ QUEUE_PREEMPTION_DISABLED, defaultVal);
}
+ public void setDisablePreemptionForPreemtionModes(String preemptionType,
boolean value) {
+setBoolean(PREEMPTION_CONFIG_PREFIX + preemptionType, value);
+ }
+
+ public boolean getDisablePreemptionForObserveOnly() {
Review Comment:
getDisablePreemptionForObserveOnly -> getPreemptionObserveOnly
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -1693,6 +1726,14 @@ public boolean getIntraQueuePreemptionDisabled(String
queue,
+ QUEUE_PREEMPTION_DISABLED, defaultVal);
}
+ public void setDisablePreemptionForPreemtionModes(String preemptionType,
boolean value) {
+setBoolean(PREEMPTION_CONFIG_PREFIX + preemptionType, value);
+ }
Review Comment:
The property observe only is a valid one, however it's not connected to
disabling preemption itself, so the naming is misleading. And observe_only is a
property of preemption, I wouldn't call it a type. Also please don't expect the
property name as a parameter, because it destroys the purpose of the setter.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration
conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
-int defaultMaxParallelAppsForQueue =
-getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
-DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
Review Comment:
It's generally a good idea to name the getters and setters exactly the same
apart from the prefix: getMaxParallelAppsForQueue
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683091#comment-17683091
]
ASF GitHub Bot commented on HADOOP-18596:
-
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093469149
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -868,35 +892,41 @@ public void testDistCpUpdateCheckFileSkip() throws
Exception {
Path source = new Path(remoteDir, "file");
Path dest = new Path(localDir, "file");
+
+Path source_0byte = new Path(remoteDir, "file_0byte");
+Path dest_0byte = new Path(localDir, "file_0byte");
dest = localFS.makeQualified(dest);
+dest_0byte = localFS.makeQualified(dest_0byte);
// Creating a source file with certain dataset.
byte[] sourceBlock = dataset(10, 'a', 'z');
// Write the dataset and as well create the target path.
-try (FSDataOutputStream out = remoteFS.create(source)) {
- out.write(sourceBlock);
- localFS.create(dest);
-}
+ContractTestUtils.createFile(localFS, dest, true, sourceBlock);
+ContractTestUtils
+.writeDataset(remoteFS, source, sourceBlock, sourceBlock.length,
+1024, true);
-verifyPathExists(remoteFS, "", source);
-verifyPathExists(localFS, "", dest);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Create 0 byte source and target files.
+ContractTestUtils.createFile(remoteFS, source_0byte, true, new byte[0]);
+ContractTestUtils.createFile(localFS, dest_0byte, true, new byte[0]);
+
+// Execute the distcp -update job.
+Job job = distCpUpdateWithFs(remoteDir, localDir, remoteFS, localFS);
// First distcp -update would normally copy the source to dest.
verifyFileContents(localFS, dest, sourceBlock);
+// Verify 1 file was skipped in the distcp -update(They 0 byte files).
Review Comment:
nit: add space after update and replace They with `the`
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -50,6 +50,7 @@
import org.apache.hadoop.tools.mapred.CopyMapper;
import org.apache.hadoop.tools.util.DistCpTestUtils;
import org.apache.hadoop.util.functional.RemoteIterators;
+import org.apache.http.annotation.Contract;
Review Comment:
where does this come from/get used?
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -913,32 +943,65 @@ public void testDistCpUpdateCheckFileSkip() throws
Exception {
long newTargetModTimeNew = modTimeSourceUpd + MODIFICATION_TIME_OFFSET;
localFS.setTimes(dest, newTargetModTimeNew, -1);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Execute the distcp -update job.
+Job updatedSourceJobOldSrc =
+distCpUpdateWithFs(remoteDir, localDir, remoteFS,
+localFS);
// File contents should remain same since the mod time for target is
// newer than the updatedSource which indicates that the sync happened
// more recently and there is no update.
verifyFileContents(localFS, dest, sourceBlock);
+// Skipped both 0 byte file and sourceFile(since mod time of target is
+// older than the source it is perceived that source is of older version
+// and we can skip it's copy).
+verifySkipAndCopyCounter(updatedSourceJobOldSrc, 2, 0);
// Subtract by an offset which would ensure enough gap for the test to
// not fail due to race conditions.
long newTargetModTimeOld =
Math.min(modTimeSourceUpd - MODIFICATION_TIME_OFFSET, 0);
localFS.setTimes(dest, newTargetModTimeOld, -1);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Execute the distcp -update job.
+Job updatedSourceJobNewSrc = distCpUpdateWithFs(remoteDir, localDir,
+remoteFS,
+localFS);
-Assertions.assertThat(RemoteIterators.toList(localFS.listFiles(dest,
true)))
-.hasSize(1);
+// Verifying the target directory have both 0 byte file and the content
+// file.
+Assertions
+.assertThat(RemoteIterators.toList(localFS.listFiles(localDir, true)))
+.hasSize(2);
// Now the copy should take place and the file contents should change
// since the mod time for target is older than the source file indicating
// that there was an up
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093469149
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -868,35 +892,41 @@ public void testDistCpUpdateCheckFileSkip() throws
Exception {
Path source = new Path(remoteDir, "file");
Path dest = new Path(localDir, "file");
+
+Path source_0byte = new Path(remoteDir, "file_0byte");
+Path dest_0byte = new Path(localDir, "file_0byte");
dest = localFS.makeQualified(dest);
+dest_0byte = localFS.makeQualified(dest_0byte);
// Creating a source file with certain dataset.
byte[] sourceBlock = dataset(10, 'a', 'z');
// Write the dataset and as well create the target path.
-try (FSDataOutputStream out = remoteFS.create(source)) {
- out.write(sourceBlock);
- localFS.create(dest);
-}
+ContractTestUtils.createFile(localFS, dest, true, sourceBlock);
+ContractTestUtils
+.writeDataset(remoteFS, source, sourceBlock, sourceBlock.length,
+1024, true);
-verifyPathExists(remoteFS, "", source);
-verifyPathExists(localFS, "", dest);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Create 0 byte source and target files.
+ContractTestUtils.createFile(remoteFS, source_0byte, true, new byte[0]);
+ContractTestUtils.createFile(localFS, dest_0byte, true, new byte[0]);
+
+// Execute the distcp -update job.
+Job job = distCpUpdateWithFs(remoteDir, localDir, remoteFS, localFS);
// First distcp -update would normally copy the source to dest.
verifyFileContents(localFS, dest, sourceBlock);
+// Verify 1 file was skipped in the distcp -update(They 0 byte files).
Review Comment:
nit: add space after update and replace They with `the`
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -50,6 +50,7 @@
import org.apache.hadoop.tools.mapred.CopyMapper;
import org.apache.hadoop.tools.util.DistCpTestUtils;
import org.apache.hadoop.util.functional.RemoteIterators;
+import org.apache.http.annotation.Contract;
Review Comment:
where does this come from/get used?
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -913,32 +943,65 @@ public void testDistCpUpdateCheckFileSkip() throws
Exception {
long newTargetModTimeNew = modTimeSourceUpd + MODIFICATION_TIME_OFFSET;
localFS.setTimes(dest, newTargetModTimeNew, -1);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Execute the distcp -update job.
+Job updatedSourceJobOldSrc =
+distCpUpdateWithFs(remoteDir, localDir, remoteFS,
+localFS);
// File contents should remain same since the mod time for target is
// newer than the updatedSource which indicates that the sync happened
// more recently and there is no update.
verifyFileContents(localFS, dest, sourceBlock);
+// Skipped both 0 byte file and sourceFile(since mod time of target is
+// older than the source it is perceived that source is of older version
+// and we can skip it's copy).
+verifySkipAndCopyCounter(updatedSourceJobOldSrc, 2, 0);
// Subtract by an offset which would ensure enough gap for the test to
// not fail due to race conditions.
long newTargetModTimeOld =
Math.min(modTimeSourceUpd - MODIFICATION_TIME_OFFSET, 0);
localFS.setTimes(dest, newTargetModTimeOld, -1);
-DistCpTestUtils
-.assertRunDistCp(DistCpConstants.SUCCESS, remoteDir.toString(),
-localDir.toString(), "-delete -update" + getDefaultCLIOptions(),
-conf);
+// Execute the distcp -update job.
+Job updatedSourceJobNewSrc = distCpUpdateWithFs(remoteDir, localDir,
+remoteFS,
+localFS);
-Assertions.assertThat(RemoteIterators.toList(localFS.listFiles(dest,
true)))
-.hasSize(1);
+// Verifying the target directory have both 0 byte file and the content
+// file.
+Assertions
+.assertThat(RemoteIterators.toList(localFS.listFiles(localDir, true)))
+.hasSize(2);
// Now the copy should take place and the file contents should change
// since the mod time for target is older than the source file indicating
// that there was an update to the source after the last sync took place.
verifyFileContents(localFS, dest, updatedSourceBlock);
+// Verifying we skipped the 0 byte file and copied the updated source
+// file (since the modification time of the new source is older than t
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683090#comment-17683090
]
ASF GitHub Bot commented on HADOOP-18596:
-
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093464483
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
oh, I didn't believe that was the case. I see the javadocs for it say we
just skip the big file tests. i'm happy
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093464483
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
oh, I didn't believe that was the case. I see the javadocs for it say we
just skip the big file tests. i'm happy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683087#comment-17683087
]
ASF GitHub Bot commented on HADOOP-18596:
-
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093465515
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
1. javadoc to include {@value}
2. distcp docs to explain the option
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093465515
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
1. javadoc to include {@value}
2. distcp docs to explain the option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683085#comment-17683085
]
ASF GitHub Bot commented on HADOOP-18596:
-
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093464483
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
oh, I didn't believe that was the case. I see the Dox say we just skipped
the big files there. If that test is happy then so am I.
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1093464483
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
oh, I didn't believe that was the case. I see the Dox say we just skipped
the big files there. If that test is happy then so am I.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18584) [NFS GW] Fix regression after netty4 migration
[ https://issues.apache.org/jira/browse/HADOOP-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-18584: - Fix Version/s: 3.3.9 > [NFS GW] Fix regression after netty4 migration > -- > > Key: HADOOP-18584 > URL: https://issues.apache.org/jira/browse/HADOOP-18584 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Two bugs were found recently. Both related to hadoop portmap utility. > (1) The ChannelHandler that can be added to a ChannelPipeline in Netty4 must > be Shareable. However, IdleStateHandler, used by Portmap is not. We need to > fix it otherwise hadoop portmap may fail to start. > (2) RpcUdpResponseStage does not get the request from client (hadoop mountd) > because of response type mismatch, and therefore it does not respond back to > client request. > This is a regression from HADOOP-11245 but I consider it not a blocker, > because most recent Linux distro supports rpcbind and there's little reason > to use Hadoop's portmap. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18584) [NFS GW] Fix regression after netty4 migration
[ https://issues.apache.org/jira/browse/HADOOP-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18584. -- Fix Version/s: 3.4.0 Resolution: Fixed > [NFS GW] Fix regression after netty4 migration > -- > > Key: HADOOP-18584 > URL: https://issues.apache.org/jira/browse/HADOOP-18584 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Two bugs were found recently. Both related to hadoop portmap utility. > (1) The ChannelHandler that can be added to a ChannelPipeline in Netty4 must > be Shareable. However, IdleStateHandler, used by Portmap is not. We need to > fix it otherwise hadoop portmap may fail to start. > (2) RpcUdpResponseStage does not get the request from client (hadoop mountd) > because of response type mismatch, and therefore it does not respond back to > client request. > This is a regression from HADOOP-11245 but I consider it not a blocker, > because most recent Linux distro supports rpcbind and there's little reason > to use Hadoop's portmap. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
slfan1989 commented on PR #5326: URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1412053772 @goiri Can you help to review this PR again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5328: YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router.
slfan1989 commented on PR #5328: URL: https://github.com/apache/hadoop/pull/5328#issuecomment-1412052529 @goiri Can you help to merge this pr into the trunk branch? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5328: YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router.
slfan1989 commented on PR #5328: URL: https://github.com/apache/hadoop/pull/5328#issuecomment-1412050766 @goiri Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5334: HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice
hadoop-yetus commented on PR #5334:
URL: https://github.com/apache/hadoop/pull/5334#issuecomment-1412039480
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 293m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5334/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 423m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5334/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5334 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux be40938ad0cd 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d7bf67a49c296203b18954f4257143bbeb846266 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5334/1/testReport/ |
| Max. process+thread count | 3189 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5334/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683009#comment-17683009
]
ASF GitHub Bot commented on HADOOP-18596:
-
hadoop-yetus commented on PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#issuecomment-1412003628
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-distcp.txt)
| hadoop-tools/hadoop-distcp: The patch generated 4 new + 14 unchanged - 0
fixed = 18 total (was 14) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 46m 9s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 173m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5308 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e05c0db28e77 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f094248e19b615310d4791e31d293b88ed6bde37 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/testReport/ |
| Max. process+thread count | 538 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/console |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
hadoop-yetus commented on PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#issuecomment-1412003628
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-distcp.txt)
| hadoop-tools/hadoop-distcp: The patch generated 4 new + 14 unchanged - 0
fixed = 18 total (was 14) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 46m 9s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 173m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5308 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e05c0db28e77 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f094248e19b615310d4791e31d293b88ed6bde37 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/testReport/ |
| Max. process+thread count | 538 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the m
[GitHub] [hadoop] hadoop-yetus commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
hadoop-yetus commented on PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1411945714
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 58s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 119m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5335 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3c39128e6dfa 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 81d122b89a699dfeaa480e62f0555b4b74f46fc1 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/1/testReport/ |
| Max. process+thread count | 564 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To uns
[GitHub] [hadoop] slfan1989 commented on pull request #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
slfan1989 commented on PR #5335: URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1411807336 @ayushtkn Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 opened a new pull request, #5335: YARN-11426. Improve YARN NodeLabel Memory Display.
slfan1989 opened a new pull request, #5335: URL: https://github.com/apache/hadoop/pull/5335 ### Description of PR JIRA: YARN-11426. Improve YARN NodeLabel Memory Display. NodeLabel displays the memory information of the cluster, directly displays the numbers, does not display the unit information, and is not easy to understand. - before improvement - after improvement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682933#comment-17682933
]
ASF GitHub Bot commented on HADOOP-18596:
-
mehakmeet commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1092974460
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
There is `TestHDFSContractDistCp.java` that does implement this suite. I
have run it before and it works successfully. Let me know if we still need to
add another one specifically for hdfs.
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
mehakmeet commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1092974460
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
There is `TestHDFSContractDistCp.java` that does implement this suite. I
have run it before and it works successfully. Let me know if we still need to
add another one specifically for hdfs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682900#comment-17682900
]
ASF GitHub Bot commented on HADOOP-18596:
-
mehakmeet commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1092876062
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
Not sure if you mean ".md" file modification or the modification to already
added java docs? I'll add some more info to the javadocs about the constant.
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
mehakmeet commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1092876062
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
Not sure if you mean ".md" file modification or the modification to already
added java docs? I'll add some more info to the javadocs about the constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
