[jira] [Resolved] (HADOOP-19072) S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-19072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-19072.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

> S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir
> -
>
> Key: HADOOP-19072
> URL: https://issues.apache.org/jira/browse/HADOOP-19072
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> on an s3a store with fs.s3a.create.performance set, speed up other operations
> *  mkdir to skip parent directory check: just do a HEAD to see if there's a 
> file at the target location



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-19218) Avoid DNS lookup while creating IPC Connection object

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-19218:
-

 Summary: Avoid DNS lookup while creating IPC Connection object
 Key: HADOOP-19218
 URL: https://issues.apache.org/jira/browse/HADOOP-19218
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Viraj Jasani


Been running HADOOP-18628 in production for quite sometime, everything works 
fine as long as DNS servers in HA are available. Upgrading single NS server at 
a time is also a common case, not problematic.

However, recently we encountered a case where 2 out of 4 NS servers went down 
(temporarily but it's a rare case). With small duration DNS cache and 2s of NS 
fallback timeout configured in resolv.conf, now any client performing DNS 
lookup can encounter 4s+ delay. This caused namenode outage as listener thread 
is single threaded and it was not able to keep up with large num of unique 
clients (in direct proportion with num of DNS resolutions every few seconds) 
initiating connection on listener port.

While having 2 out of 4 DNS servers offline is rare case and NS fallback 
settings could also be improved, it is important to note that we don't need to 
perform DNS resolution for every new connection if the intention is to improve 
the insights into VersionMistmatch errors thrown by the server.

The proposal is the delay the DNS resolution until the server throws the error 
for incompatible header or version mismatch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Full builds not reporting status back to Github PRs

[Viraj Jasani]

Most important, yetus report is usually not available in such cases. e.g.
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6830/2/Yetus_20Report/


On Tue, May 28, 2024 at 2:05 PM Viraj Jasani  wrote:

> Some of the recent PRs that require full build runs, it seems that the
> builds are not able to get the build results posted to the PRs. Just wanted
> to check if anyone has any ideas reg this.
>
> Sample recent PRs:
>
>- https://github.com/apache/hadoop/pull/6830
>- https://github.com/apache/hadoop/pull/6247
>- https://github.com/apache/hadoop/pull/6842
>
>
>

Full builds not reporting status back to Github PRs

[Viraj Jasani]

Some of the recent PRs that require full build runs, it seems that the
builds are not able to get the build results posted to the PRs. Just wanted
to check if anyone has any ideas reg this.

Sample recent PRs:

   - https://github.com/apache/hadoop/pull/6830
   - https://github.com/apache/hadoop/pull/6247
   - https://github.com/apache/hadoop/pull/6842

[jira] [Created] (HADOOP-19146) noaa-cors-pds bucket access with global endpoint fails

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-19146:
-

 Summary: noaa-cors-pds bucket access with global endpoint fails
 Key: HADOOP-19146
 URL: https://issues.apache.org/jira/browse/HADOOP-19146
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Affects Versions: 3.4.0
Reporter: Viraj Jasani


All tests accessing noaa-cors-pds use us-east-1 region, as configured at bucket 
level. If global endpoint is configured (e.g. us-west-2), they fail to access 
to bucket.

 

Sample error:
{code:java}
org.apache.hadoop.fs.s3a.AWSRedirectException: Received permanent redirect 
response to region [us-east-1].  This likely indicates that the S3 region 
configured in fs.s3a.endpoint.region does not match the AWS region containing 
the bucket.: null (Service: S3, Status Code: 301, Request ID: PMRWMQC9S91CNEJR, 
Extended Request ID: 
6Xrg9thLiZXffBM9rbSCRgBqwTxdLAzm6OzWk9qYJz1kGex3TVfdiMtqJ+G4vaYCyjkqL8cteKI/NuPBQu5A0Q==)
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:253)
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4041)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3947)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$26(S3AFileSystem.java:3924)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3922)
    at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
    at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
    at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$globStatus$35(S3AFileSystem.java:4956)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
    at 
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735)
    at 
org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:4949)
    at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:313)
    at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:281)
    at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:445)
    at 
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:311)
    at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:328)
    at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:201)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1677)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1674)
 {code}
{code:java}
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: 
S3, Status Code: 301, Request ID: PMRWMQC9S91CNEJR, Extended Request ID: 
6Xrg9thLiZXffBM9rbSCRgBqwTxdLAzm6OzWk9qYJz1kGex3TVfdiMtqJ+G4vaYCyjkqL8cteKI/NuPBQu5A0Q==)
    at 
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at 
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at 
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at 
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at 
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:93)
    at 
software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
    ...
    ...
    ...
    at 
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53

[jira] [Created] (HADOOP-19066) AWS SDK V2 - Enabling FIPS should be allowed with central endpoint

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-19066:
-

 Summary: AWS SDK V2 - Enabling FIPS should be allowed with central 
endpoint
 Key: HADOOP-19066
 URL: https://issues.apache.org/jira/browse/HADOOP-19066
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.5.0, 3.4.1
Reporter: Viraj Jasani


FIPS support can be enabled by setting "fs.s3a.endpoint.fips". Since the SDK 
considers overriding endpoint and enabling fips as mutually exclusive, we fail 
fast if fs.s3a.endpoint is set with fips support (details on HADOOP-18975).

Now, we no longer override SDK endpoint for central endpoint since we enable 
cross region access (details on HADOOP-19044) but we would still fail fast if 
endpoint is central and fips is enabled.

Changes proposed:
 * S3A to fail fast only if FIPS is enabled and non-central endpoint is 
configured.
 * Tests to ensure S3 bucket is accessible with default region us-east-2 with 
cross region access (expected with central endpoint).
 * Document FIPS support with central endpoint on connecting.html.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-19023) ITestS3AConcurrentOps#testParallelRename intermittent timeout failure

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-19023:
-

 Summary: ITestS3AConcurrentOps#testParallelRename intermittent 
timeout failure
 Key: HADOOP-19023
 URL: https://issues.apache.org/jira/browse/HADOOP-19023
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani


Need to configure higher timeout for the test.

 
{code:java}
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 256.281 
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR] 
testParallelRename(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)  Time 
elapsed: 72.565 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: Writing Object on 
fork-0005/test/testParallelRename-source0: 
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution 
did not complete before the specified timeout configuration: 15000 millis
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at 
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:214)
at 
org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:532)
at 
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:620)
at 
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at 
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at 
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at 
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at 
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.ApiCallTimeoutException: 
Client execution did not complete before the specified timeout configuration: 
15000 millis
at 
software.amazon.awssdk.core.exception.ApiCallTimeoutException$BuilderImpl.build(ApiCallTimeoutException.java:97)
at 
software.amazon.awssdk.core.exception.ApiCallTimeoutException.create(ApiCallTimeoutException.java:38)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.generateApiCallTimeoutException(ApiCallTimeoutTrackingStage.java:151)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:139)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at 
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at 
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at 
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at 
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.ja

[jira] [Created] (HADOOP-19022) ITestS3AConfiguration#testRequestTimeout failure

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-19022:
-

 Summary: ITestS3AConfiguration#testRequestTimeout failure
 Key: HADOOP-19022
 URL: https://issues.apache.org/jira/browse/HADOOP-19022
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani


"fs.s3a.connection.request.timeout" should be specified in milliseconds as per
{code:java}
Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO); 
{code}
The test fails consistently because it sets 120 ms timeout which is less than 
15s (min network operation duration), and hence gets reset to 15000 ms based on 
the enforcement.

 
{code:java}
[ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  
Time elapsed: 0.016 s  <<< FAILURE!
java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is 
different than what AWS sdk configuration uses internally expected:<12> but 
was:<15000>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at 
org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18959) Use builder for prefetch CachingBlockManager

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18959:
-

 Summary: Use builder for prefetch CachingBlockManager
 Key: HADOOP-18959
 URL: https://issues.apache.org/jira/browse/HADOOP-18959
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani


Some of the recent changes (HADOOP-18399, HADOOP-18291, HADOOP-18829 etc) have 
added more params for prefetch CachingBlockManager c'tor to process read/write 
block requests. They have added too many params and more are likely to be 
introduced later. We should use builder pattern to pass params.

This would also help consolidating required prefetch params into one single 
place within S3ACachingInputStream, from scattered locations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-18829) s3a prefetch LRU cache eviction metric

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-18829.
---
Fix Version/s: 3.4.0
   3.3.9
 Hadoop Flags: Reviewed
   Resolution: Fixed

> s3a prefetch LRU cache eviction metric
> --
>
> Key: HADOOP-18829
> URL: https://issues.apache.org/jira/browse/HADOOP-18829
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Follow-up from HADOOP-18291:
> Add new IO statistics metric to capture s3a prefetch LRU cache eviction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18918) ITestS3GuardTool fails if SSE encryption is used

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18918:
-

 Summary: ITestS3GuardTool fails if SSE encryption is used
 Key: HADOOP-18918
 URL: https://issues.apache.org/jira/browse/HADOOP-18918
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, test
Affects Versions: 3.3.6
Reporter: Viraj Jasani


{code:java}
[ERROR] Tests run: 15, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 25.989 
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[ERROR] 
testLandsatBucketRequireUnencrypted(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool)
  Time elapsed: 0.807 s  <<< ERROR!
46: Bucket s3a://landsat-pds: required encryption is none but actual encryption 
is DSSE-KMS
    at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.exitException(S3GuardTool.java:915)
    at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.badState(S3GuardTool.java:881)
    at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:511)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:283)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:963)
    at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardToolTestHelper.runS3GuardCommand(S3GuardToolTestHelper.java:147)
    at 
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.run(AbstractS3GuardToolTestBase.java:114)
    at 
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool.testLandsatBucketRequireUnencrypted(ITestS3GuardTool.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
    at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
    at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:750)
 {code}
Since landsat requires none encryption, the test should be skipped for any 
encryption algorithm.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18832) Upgrade aws-java-sdk to 1.12.499+

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18832:
-

 Summary: Upgrade aws-java-sdk to 1.12.499+
 Key: HADOOP-18832
 URL: https://issues.apache.org/jira/browse/HADOOP-18832
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Viraj Jasani


aws sdk versions < 1.12.499 uses a vulnerable version of netty and hence 
showing up in security CVE scans (CVE-2023-34462). The safe version for netty 
is 4.1.94.Final and this is used by aws-java-adk:1.12.499+



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-18207 hadoop-logging module about to land

[Viraj Jasani]

Thank you Wei-Chiu for the thread and extensive help with reviews! Thank
you Ayush for responding to the thread!
Let me try to address some points.

Please pardon my ignorance if I am not supposed to respond to any of the
questions.

> Regarding this entire activity including the parent tickets: Do we have
any dev list agreement for this?

HADOOP-16206  was
created back in Mar, 2019 and there has been tons of discussion on the Jira
since then. Duo is an expert and he has also worked with our esteemed Log4j
community to introduce changes that promise great benefits for both hbase
and hadoop projects (for instance, [1]). He has laid out the plan to tackle
the whole migration, one small piece at a time and there has been enough
agreement on the Jira from Hadoop experts, some of Log4j community members
also chimed in and provided their feedbacks, and it has been agreed upon to
proceed with Duo's proposed plan and integrate the changes into the trunk.
This will enable us to stabilize the work gradually over time.
The Jira has received many interactions over the past few years.


> What incompatibilities have been introduced till now for this and what
are planned.

Let me list down what has been done so far, that might be easier to discuss:


   - HADOOP-18206  removed
   commons-logging references, the project is no longer under any active
   development cycle (last release on 2014
   https://github.com/apache/commons-logging/tags), and without this
   cleanup, it becomes very difficult to chase log4j changes. No direct
   incompatibility involved.
   - HADOOP-18653 
follow-up
   to ensure we use slf4j log4j adaptor to ensure slf4j is in the classpath
   before we update loglevel (dynamically change log level using servlet). No
   incompatibility introduced.
   - HADOOP-18648  kms
   log4j properties should not be loaded dynamically as this is no longer
   supported by log4j2, instead use HADOOP_OPTS to provide log4j properties
   location. No incompatibility introduced.
   - HADOOP-18654 
TaskLogAppender
   is not being used, remove it. It was marked IA.Private and IS.Unstable. No
   incompatibility introduced.
   - HADOOP-18669 
remove Log4Json
   Layout as it is more suitable to be part of Log4j project rather than
   Hadoop, it's not being used anywhere. Each appender that we maintain, we
   pay for its maintenance cost. No incompatibility introduced.
   - HADOOP-18649  CLA
   and CLRA appenders to be replaced with log4j RFA appender. Both CLA and
   CLRA have been our custom appenders and they both provide the same
   capabilities as RFA hence their maintenance in our project would come with
   cost for any future upgrades of log4j. This has also been agreed upon on
   the parent Jira way before the work started.
   - HADOOP-18631  Migrate
   dynamic async appenders to log4j properties. This is *an incompatible
   change* because we replace "hadoop site configs" with "log4j
   properties". We are not losing out on our capability to generate async logs
   for namenode audit, but the way to configure it is now different. The
   release notes have been updated to reflect the same. For log4j upgrade, we
   don't have a choice here, log4j2 only supports async loggers as the
   configuration, not as programmatically loaded appenders. log4j properties
   to configure are provided on
   
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties#L64-L81


As for the current task that introduced hadoop-logging module HADOOP-18207
, we don't have any
incompatibility yet because our direct usage of log4j APIs and custom
appenders have not been marked IA.Public.

The major incompatibility is going to be introduced when we add log4j2 in
our classpath and remove log4j1 from our dependencies. This work has not
yet begun and it's going to take a while.


> What does this activity bring up for the downstream folks adapting this?
Upgrading Hadoop is indeed important for a lot of projects

Downstream projects should expect to no longer get log4j1 as a transitive
dependency from hadoop, instead they would get log4j2 as transitive
dependency (only after the whole upgrade is done, log4j2 upgrade has not
even started as I mentioned above :)).

This brings an interesting question: why do we need this upgrade? For us,
almost all of hadoop ecosystem projects that we use have migrated to
log4j2, and when we keep common thirdparty dependencies to be used by all
hadoop downstreamers, we can still not use log4j2 because hado

[jira] [Created] (HADOOP-18829) s3a prefetch LRU cache eviction metric

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18829:
-

 Summary: s3a prefetch LRU cache eviction metric
 Key: HADOOP-18829
 URL: https://issues.apache.org/jira/browse/HADOOP-18829
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Follow-up from HADOOP-18291:

Add new IO statistics metric to capture s3a prefetch LRU cache eviction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18809) s3a prefetch read/write file operations should guard channel close

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18809:
-

 Summary: s3a prefetch read/write file operations should guard 
channel close
 Key: HADOOP-18809
 URL: https://issues.apache.org/jira/browse/HADOOP-18809
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As per Steve's suggestion from s3a prefetch LRU cache,

s3a prefetch disk based cache file read and write operations should guard 
against close of FileChannel and WritableByteChannel, close them even if 
read/write operations throw IOException.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Viraj Jasani]

+1 (non-binding)

* Signature: ok
* Checksum: ok
* Rat check (1.8.0_362): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_362): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_362): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Brought up pseudo-distributed mode cluster with hdfs, zk 3.7 and hbase 2.5
* Basic RBF tests look good
* S3A tests with input stream prefetch enabled: looks good


On Sun, Jun 18, 2023 at 5:53 PM Wei-Chiu Chuang  wrote:

> I am inviting anyone to try and vote on this release candidate.
>
> Note:
> This is exactly the same as RC0, except the CHANGELOG.
>
> The RC is available at:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
>
> Git tag: release-3.3.6-RC1
> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
>
> This is a relatively small release (by Hadoop standard) containing about
> 120 commits.
> Please give it a try, this RC vote will run for 7 days.
>
>
> Feature highlights:
>
> SBOM artifacts
> 
> Starting from this release, Hadoop publishes Software Bill of Materials
> (SBOM) using
> CycloneDX Maven plugin. For more information about SBOM, please go to
> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
>
> HDFS RBF: RDBMS based token storage support
> 
> HDFS Router-Router Based Federation now supports storing delegation tokens
> on MySQL,
> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> which improves token operation through over the original Zookeeper-based
> implementation.
>
>
> New File System APIs
> 
> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
> number of
> HDFS-specific APIs to Hadoop Common to make it possible for certain
> applications that
> depend on HDFS semantics to run on other Hadoop compatible file systems.
>
> In particular, recoverLease() and isFileClosed() are exposed through
> LeaseRecoverable
> interface. While setSafeMode() is exposed through SafeMode interface.
>

[jira] [Created] (HADOOP-18740) s3a prefetch cache blocks should be accessed by RW locks

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18740:
-

 Summary: s3a prefetch cache blocks should be accessed by RW locks
 Key: HADOOP-18740
 URL: https://issues.apache.org/jira/browse/HADOOP-18740
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


In order to implement LRU or LFU based cache removal policies for s3a 
prefetched cache blocks, it is important for all cache reader threads to 
acquire read lock and similarly cache file removal mechanism (fs close or cache 
eviction) to acquire write lock before accessing the files.

As we maintain the block entries in an in-memory map, we should be able to 
introduce read-write lock per cache file entry, we don't need coarse-grained 
lock shared by all entries.

 

This is a prerequisite to HADOOP-18291.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17612) Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-17612.
---
Resolution: Fixed

> Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0
> ---
>
> Key: HADOOP-17612
> URL: https://issues.apache.org/jira/browse/HADOOP-17612
> Project: Hadoop Common
>  Issue Type: Task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Let's upgrade Zookeeper and Curator to 3.6.3 and 5.2.0 respectively.
> Curator 5.2 also supports Zookeeper 3.5 servers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-17612) Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HADOOP-17612:
---

Reopening to update the resolution

> Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0
> ---
>
> Key: HADOOP-17612
> URL: https://issues.apache.org/jira/browse/HADOOP-17612
> Project: Hadoop Common
>  Issue Type: Task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Let's upgrade Zookeeper and Curator to 3.6.3 and 5.2.0 respectively.
> Curator 5.2 also supports Zookeeper 3.5 servers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

[Viraj Jasani]

IIRC some of the ongoing major dependency upgrades (log4j 1 to 2, jersey 1
to 2 and junit 4 to 5) are blockers for java 11 compile + test stability.


On Tue, Mar 28, 2023 at 4:55 AM Steve Loughran 
wrote:

>  Now that hadoop 3.3.5 is out, i want to propose something new
>
> we switch branch-3.3 and trunk to being java11 only
>
>
>1. java 11 has been out for years
>2. oracle java 8 is no longer available under "premier support"; you
>can't really get upgrades
>https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>3. openJDK 8 releases != oracle ones, and things you compile with them
>don't always link to oracle java 8 (some classes in java.nio have added
>more overrides)
>4. more and more libraries we want to upgrade to/bundle are java 11 only
>5. moving to java 11 would cut our yetus build workload in half, and
>line up for adding java 17 builds instead.
>
>
> I know there are some outstanding issues still in
> https://issues.apache.org/jira/browse/HADOOP-16795 -but are they blockers?
> Could we just move to java11 and enhance at our leisure, once java8 is no
> longer a concern.
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Viraj Jasani]

+1 (non-binding)

* Signature/Checksum: ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Containerized deployments:
* Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
* Deployed and started JHS, RM, NM
* Hbase, hdfs CRUD looks good
* Sample RowCount MapReduce job looks good

* S3A tests with scale profile looks good


On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran 
wrote:

> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>
> The git tag is release-3.3.5-RC3, commit 706d88266ab
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
> * and compared to the previous RC, all the major changes are
>   HDFS issues.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

[jira] [Created] (HADOOP-18669) Remove Log4Json Layout

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18669:
-

 Summary: Remove Log4Json Layout
 Key: HADOOP-18669
 URL: https://issues.apache.org/jira/browse/HADOOP-18669
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Log4Json extends org.apache.log4j.Layout to provide log layout for Json. This 
utility is not being used anywhere in Hadoop. It is IA.Private (by default).

Log4j2 has introduced drastic changes to the Layout. It also converted it as an 
interface. Log4j2 also has JsonLayout, it provides options like Pretty vs. 
compact JSON, Encoding UTF-8 or UTF-16, Complete well-formed JSON vs. fragment 
JSON, addition of custom fields into generated JSON

[https://github.com/apache/logging-log4j2/blob/2.x/log4j-core/src/main/java/org/apache/logging/log4j/core/layout/JsonLayout.java]

 

This utility is more suitable to be part of log4j project rather than hadoop 
because the maintenance cost in hadoop would be higher with any more upgrades 
introducing changes to the Layout format.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-18631) Migrate Async appenders to log4j properties

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-18631.
---
Resolution: Fixed

> Migrate Async appenders to log4j properties
> ---
>
> Key: HADOOP-18631
> URL: https://issues.apache.org/jira/browse/HADOOP-18631
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Before we can upgrade to log4j2, we need to migrate async appenders that we 
> add "dynamically in the code" to the log4j.properties file. Instead of using 
> core/hdfs site configs, log4j properties or system properties should be used 
> to determine if the given logger should use async appender.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-18631) Migrate Async appenders to log4j properties

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HADOOP-18631:
---

> Migrate Async appenders to log4j properties
> ---
>
> Key: HADOOP-18631
> URL: https://issues.apache.org/jira/browse/HADOOP-18631
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Before we can upgrade to log4j2, we need to migrate async appenders that we 
> add "dynamically in the code" to the log4j.properties file. Instead of using 
> core/hdfs site configs, log4j properties or system properties should be used 
> to determine if the given logger should use async appender.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18668) Path capability probe for truncate is only honored by RawLocalFileSystem

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18668:
-

 Summary: Path capability probe for truncate is only honored by 
RawLocalFileSystem
 Key: HADOOP-18668
 URL: https://issues.apache.org/jira/browse/HADOOP-18668
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


FileSystem#hasPathCapability returns true for probing 
"fs.capability.paths.truncate" only by RawLocalFileSystem. It should be honored 
by all file system implementations that support truncate.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18654) Remove unused custom appender TaskLogAppender

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18654:
-

 Summary: Remove unused custom appender TaskLogAppender
 Key: HADOOP-18654
 URL: https://issues.apache.org/jira/browse/HADOOP-18654
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


TaskLogAppender is no longer being used in codebase. The only past references 
we have are from old releasenotes (HADOOP-7308, MAPREDUCE-3208, MAPREDUCE-2372, 
HADOOP-1355).

Before we migrate to log4j2, it would be good to remove TaskLogAppender.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18653) LogLevel servlet to determine log impl before using setLevel

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18653:
-

 Summary: LogLevel servlet to determine log impl before using 
setLevel
 Key: HADOOP-18653
 URL: https://issues.apache.org/jira/browse/HADOOP-18653
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


LogLevel GET API is used to set log level for a given class name dynamically. 
While we have cleaned up the commons-logging references, it would be great to 
determine whether slf4j log4j adapter is in the classpath before allowing 
client to set the log level.

Proposed changes:
 * Use slf4j logger factory to get the log reference for the given class name
 * Use generic utility to identify if the slf4j log4j adapter is in the 
classpath before using log4j API to update the log level
 * If the log4j adapter is not in the classpath, report error in the output



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Viraj Jasani]

A minor update on ITestS3AConcurrentOps#testParallelRename

I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).


On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran 
wrote:

> On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> 
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this  release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >>  - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >>  - mvn clean install  -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >>  - mvn clean package  -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, insta

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Viraj Jasani]

While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job

* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale

ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.


On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
>1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
>issues in maven 3.9.0)
>4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>

[jira] [Created] (HADOOP-18649) CLA and CRLA appenders to be replaced with RFA

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18649:
-

 Summary: CLA and CRLA appenders to be replaced with RFA
 Key: HADOOP-18649
 URL: https://issues.apache.org/jira/browse/HADOOP-18649
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


ContainerLogAppender and ContainerRollingLogAppender both have quite similar 
functionality as RollingFileAppender. Maintenance of custom appenders for 
Log4J2 is costly when there is very minor difference in comparison with 
built-in appender provided by Log4J. 

The goal of this sub-task is to replace both ContainerLogAppender and 
ContainerRollingLogAppender custom appenders with RollingFileAppender without 
changing any system properties already being used to determine file name, file 
size, backup index, pattern layout properties etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18648) Avoid loading kms log4j properties dynamically by KMSWebServer

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18648:
-

 Summary: Avoid loading kms log4j properties dynamically by 
KMSWebServer
 Key: HADOOP-18648
 URL: https://issues.apache.org/jira/browse/HADOOP-18648
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Log4j2 does not support loading of log4j properties (/xml/json/yaml) 
dynamically by applications. It no longer supports overriding the loading of 
properties using "log4j.defaultInitOverride" the way log4j1 does.

For KMS, instead of loading the properties file dynamically, we should add the 
log4j properties file as part of HADOOP_OPTS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18645) Provide keytab file key name with ServiceStateException

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18645:
-

 Summary: Provide keytab file key name with ServiceStateException
 Key: HADOOP-18645
 URL: https://issues.apache.org/jira/browse/HADOOP-18645
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Viraj Jasani
Assignee: Viraj Jasani


 
{code:java}
util.ExitUtil - Exiting with status 1: 
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Running 
in secure mode, but config doesn't have a keytab
1: org.apache.hadoop.service.ServiceStateException: java.io.IOException: 
Running in secure mode, but config doesn't have a keytab
  at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:264)
..
..
 {code}
 

 

When multiple downstreamers use different configs to present the same keytab 
file, if one of the config key gets missing or overridden as part of config 
generators, it becomes bit confusing for operators to realize which config is 
missing for a particular service, especially when keytab file value is already 
present with different config.

It would be nice to report config key with the stacktrace error message.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Socket timeout settings

[Viraj Jasani]

We have a specific environment where we need to harmonize socket connection
timeouts for all Hadoop daemons and some downstreamers too. While reviewing
the socket connection timeouts set in NetUtils, UrlConnection
(HttpURLConnection), I compiled a list of the following configurations:

   - ipc.client.connect.timeout
   - dfs.client.socket-timeout
   - dfs.datanode.socket.write.timeout
   - dfs.client.fsck.connect.timeout
   - dfs.client.fsck.read.timeout
   - dfs.federation.router.connect.timeout
   - dfs.qjournal.http.open.timeout.ms
   - dfs.qjournal.http.read.timeout.ms
   - dfs.checksum.ec.socket-timeout
   - hadoop.security.kms.client.timeout
   - mapreduce.reduce.shuffle.connect.timeout
   - mapreduce.reduce.shuffle.read.timeout


Moreover, although “dfs.datanode.socket.reuse.keepalive” does not indicate
a direct socket timeout, we set it as SocketOptions#SO_TIMEOUT if
opsProcessed != 0 (to block read on InputStream only for this timeout,
beyond which it would result in SocketTimeoutException). Similarly,
“ipc.ping.interval” and “ipc.client.rpc-timeout.ms” are also used to set
SocketOptions#SO_TIMEOUT on the socket.

It's possible that I may have missed some socket timeout configs in the
above list. If anyone could provide feedback on this list or suggest any
missing configs, it would be greatly appreciated.

[jira] [Created] (HADOOP-18631) Migrate Async appenders to log4j properties

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18631:
-

 Summary: Migrate Async appenders to log4j properties
 Key: HADOOP-18631
 URL: https://issues.apache.org/jira/browse/HADOOP-18631
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Before we can upgrade to log4j2, we need to migrate async appenders that we add 
"dynamically in the code" to the log4j.properties file. Instead of using 
core/hdfs site configs, whether to use async appenders should be decided based 
on system properties that log4j properties can derive value from.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18628) Server connection should log host name before returning VersionMismatch error

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18628:
-

 Summary: Server connection should log host name before returning 
VersionMismatch error
 Key: HADOOP-18628
 URL: https://issues.apache.org/jira/browse/HADOOP-18628
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Viraj Jasani
Assignee: Viraj Jasani


In env with dynamically changing IP addresses, debugging issue with the logs 
with only IP address becomes a bit difficult at times.
{code:java}
2023-02-08 23:26:50,112 WARN  [Socket Reader #1 for port 8485] ipc.Server - 
Incorrect RPC Header length from {IPV4}:36556 expected length: 
java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] got length: 
java.nio.HeapByteBuffer[pos=0 lim=4 cap=4] {code}
It would be better to log full hostname for the given IP address rather than 
only IP address.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18620) Avoid using grizzly-http classes

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18620:
-

 Summary: Avoid using grizzly-http classes
 Key: HADOOP-18620
 URL: https://issues.apache.org/jira/browse/HADOOP-18620
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As discussed on the parent Jira HADOOP-15984, we do not have any 
grizzly-http-servlet version available that uses Jersey 2 dependencies. 

version 2.4.4 contains Jersey 1 artifacts: 
[https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/2.4.4/grizzly-http-servlet-2.4.4.pom]

The next higher version available is 3.0.0-M1 and it contains Jersey 3 
artifacts: 
[https://repo1.maven.org/maven2/org/glassfish/grizzly/grizzly-http-servlet/3.0.0-M1/grizzly-http-servlet-3.0.0-M1.pom]

 

Moreover, we do not use grizzly-http-* modules extensively. We use them only 
for few tests such that we don't have to implement all the methods of 
HttpServletResponse for our custom test classes.

We should get rid of grizzly-http-servlet, grizzly-http and grizzly-http-server 
artifacts of org.glassfish.grizzly and rather implement HttpServletResponse 
directly to avoid having to depend on grizzly upgrades as part of overall 
Jersey upgrade.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18592) Sasl connection failure should log remote address

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18592:
-

 Summary: Sasl connection failure should log remote address
 Key: HADOOP-18592
 URL: https://issues.apache.org/jira/browse/HADOOP-18592
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.3.4
Reporter: Viraj Jasani
Assignee: Viraj Jasani


If Sasl connection fails with some generic error, we miss logging remote server 
that the client was trying to connect to.

Sample log:
{code:java}
2023-01-12 00:22:28,148 WARN  [20%2C1673404849949,1] ipc.Client - Exception 
encountered while connecting to the server 
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at 
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
    at 
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
    at 
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
    at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
...
... {code}
We should log the remote server address.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5

[Viraj Jasani]

-0 (non-binding)

Output of hadoop-vote.sh:

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Manual testing on local mini cluster:
* Basic CRUD tests on Hdfs looks good
* Sample MapReduce job looks good
* S3A tests look good with scale profile (ITestS3AContractUnbuffer is
flaky, but when run individually, it passes)

Full build with all modules UT results for branch-3.3.5 latest HEAD are
available on
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-3.3.5-java8-linux-x86_64/

>From the above build, there are some consistently failing tests, out of
which only TestDataNodeRollingUpgrade passed locally, whereas rest of the
tests are consistently failing locally as well, we might want to fix (or
ignore, if required) them:

org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshot#testSnapshotsOnErasureCodingDirAfterNNRestart
org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart#testFileLengthWithHSyncAndClusterRestartWithOutDNsRegister
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart2
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryWithRenameAfterNameNodeRestart
org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade#testWithLayoutChangeAndFinalize
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot#testSnapshotOpsOnRootReservedPath
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap#testReadRenamedSnapshotFileWithCheckpoint
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion#testApplyEditLogForDeletion



On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>
> Given the time of year it's a bit unrealistic to run a 5 day vote and
> expect people to be able to test it thoroughly enough to make this the one
> we can ship.
>
> What we would like is for anyone who can to verify the tarballs, and test
> the binaries, especially anyone who can try the arm64 binaries. We've got
> the building of those done and now the build file will incorporate them
> into the release -but neither of us have actually tested it yet. Maybe I
> should try it on my pi400 over xmas.
>
> The maven artifacts are up on the apache staging repo -they are the ones
> from x86 build. Building and testing downstream apps will be incredibly
> helpful.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>
> The git tag is release-3.3.5-RC0, commit 3262495904d
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positive.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
>
>
> Please try the release and vote on it, even though i don't know what is a
> good timeline here...i'm actually going on holiday in early jan. Mukund is
> around and so can drive the process while I'm offline.
>
> Assuming we do have another iteration, the RC1 will not be before mid jan
> for that reason
>
> Steve (and mukund)
>

[jira] [Created] (HADOOP-18466) Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18466:
-

 Summary: Limit the findbugs suppression IS2_INCONSISTENT_SYNC to 
S3AFileSystem field
 Key: HADOOP-18466
 URL: https://issues.apache.org/jira/browse/HADOOP-18466
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field 
futurePool to avoid letting it discover other synchronization bugs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18455) s3a prefetching Executor should be closed

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18455:
-

 Summary: s3a prefetching Executor should be closed
 Key: HADOOP-18455
 URL: https://issues.apache.org/jira/browse/HADOOP-18455
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Viraj Jasani
Assignee: Viraj Jasani


This is the follow-up work for HADOOP-18186. The new executor service we use 
for s3a prefetching should be closed while shutting down the file system.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-18186.
---
Resolution: Fixed

> s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
> ---
>
> Key: HADOOP-18186
> URL: https://issues.apache.org/jira/browse/HADOOP-18186
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Use SemaphoredDelegatingExecutor for each to stream to submit work, if 
> possible, for better fairness in processes with many streams.
> this also takes a DurationTrackerFactory to count how long was spent in the 
> queue, something we would want to know



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HADOOP-18186:
---

Re-opening for an addendum

> s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
> ---
>
> Key: HADOOP-18186
> URL: https://issues.apache.org/jira/browse/HADOOP-18186
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Use SemaphoredDelegatingExecutor for each to stream to submit work, if 
> possible, for better fairness in processes with many streams.
> this also takes a DurationTrackerFactory to count how long was spent in the 
> queue, something we would want to know



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18435) Remove usage of fs.s3a.executor.capacity

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18435:
-

 Summary: Remove usage of fs.s3a.executor.capacity
 Key: HADOOP-18435
 URL: https://issues.apache.org/jira/browse/HADOOP-18435
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Viraj Jasani
Assignee: Viraj Jasani


When s3guard was part of s3a, DynamoDBMetadataStore was the only consumer of 
StoreContext that used throttled executor provided by StoreContext, which 
internally uses fs.s3a.executor.capacity to determine executor capacity for 
SemaphoredDelegatingExecutor. With the removal of s3guard from s3a, we should 
also remove fs.s3a.executor.capacity and it's usages as it's no longer being 
used by any StoreContext consumers. The config's existence and its description 
can be really confusing for the users.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18403) Fix FileSystem leak in ITestS3AAWSCredentialsProvider

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18403:
-

 Summary: Fix FileSystem leak in ITestS3AAWSCredentialsProvider
 Key: HADOOP-18403
 URL: https://issues.apache.org/jira/browse/HADOOP-18403
 Project: Hadoop Common
  Issue Type: Test
Reporter: Viraj Jasani
Assignee: Viraj Jasani


ITestS3AAWSCredentialsProvider#testAnonymousProvider has FileSystem leak that 
should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18397) Shutdown AWSSecurityTokenService when it's resources are no longer in use

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18397:
-

 Summary: Shutdown AWSSecurityTokenService when it's resources are 
no longer in use
 Key: HADOOP-18397
 URL: https://issues.apache.org/jira/browse/HADOOP-18397
 Project: Hadoop Common
  Issue Type: Task
  Components: fs/s3
Reporter: Viraj Jasani
Assignee: Viraj Jasani


AWSSecurityTokenService resources can be released whenever they are no longer 
in use. The documentation of AWSSecurityTokenService#shutdown says while it is 
not important for client to compulsorily shutdown the token service, client can 
definitely perform early release whenever client no longer requires token 
service resources. We achieve this by making STSClient closable, so we can 
certainly utilize it in all places where it's suitable.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HADOOP-18303.
---
Resolution: Won't Fix

> Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
> --
>
> Key: HADOOP-18303
> URL: https://issues.apache.org/jira/browse/HADOOP-18303
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from 
> both hadoop-client-runtime and hadoop-client-minicluster. This has caused 
> issues for downstreamers e.g. 
> [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions.
> We should put the shading back in hadoop-client-runtime to fix CNFE issues 
> for downstreamers.
> cc [~ayushsaxena] [~pan3793] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18303:
-

 Summary: Remove shading exclusion of javax.ws.rs-api from 
hadoop-client-runtime
 Key: HADOOP-18303
 URL: https://issues.apache.org/jira/browse/HADOOP-18303
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from both 
hadoop-client-runtime and hadoop-client-minicluster. This has caused issues for 
downstreamers e.g. [https://github.com/apache/incubator-kyuubi/issues/2904], 
more discussions.

We should put the shading back in hadoop-client-runtime to fix CNFE issues for 
downstreamers.

cc [~ayushsaxena] [~pan3793] 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18288) Total requests and total requests per sec served by RPC servers

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18288:
-

 Summary: Total requests and total requests per sec served by RPC 
servers
 Key: HADOOP-18288
 URL: https://issues.apache.org/jira/browse/HADOOP-18288
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Viraj Jasani
Assignee: Viraj Jasani


RPC Servers provide bunch of useful information like num of open connections, 
slow requests, num of in-progress handlers, RPC processing time, queue time 
etc, however so far it doesn't provide accumulation of all requests as well as 
current snapshot of requests per second served by the server. Exposing them 
would benefit from operational viewpoint in identifying how busy the servers 
have been and how much load they are currently serving in the presence of 
cluster wide high load.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Viraj Jasani]

Sounds good, yes they both are not blocker issues for release anyways
(hence +0), just wanted to get more insights. Thanks for the detailed
clarifications Ayush!

Changing my vote to +1 (non-binding).


On Fri, May 27, 2022 at 1:24 PM Ayush Saxena  wrote:

> The checksum stuff was addressed in HADOOP-16985, so that filename stuff
> is sorted only post 3.3.x
> BTW it is a known issue:
>
> https://issues.apache.org/jira/browse/HADOOP-16494?focusedCommentId=16927236&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16927236
>
> Must not be a blocker for us
>
> The RAT check failing with dependency issue. That also should work post
> 3.3.x because there is no Hadoop-maven-plugin dependency in Hadoop-yarn-api
> module post 3.3.x, HADOOP-16560 removed it.
> Ref:
> https://github.com/apache/hadoop/pull/1496/files#diff-f5d219eaf211871f9527ae48da59586e7e9958ea7649de74a1393e599caa6dd6L121-R122
>
> So, that is why the RAT check passes for 3.3.x+ without the need of this
> module. Committing HADOOP-16663, should solve this though.(I haven't tried
> though, just by looking at the problem)
>
> Good to have patches, but doesn't look like blockers to me. kind of build
> related stuffs only, nothing bad with our core Hadoop code.
>
> -Ayush
>
> On Sat, 28 May 2022 at 01:04, Viraj Jasani  wrote:
>
>> +0 (non-binding),
>>
>> * Signature/Checksum looks good, though I am not sure where
>> "target/artifacts" is coming from for the tars, here is the diff (this was
>> the case for 2.10.1 as well but checksum was correct):
>>
>> 1c1
>> < SHA512 (hadoop-2.10.2-site.tar.gz) =
>>
>> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
>> ---
>> > SHA512 (target/artifacts/hadoop-2.10.2-site.tar.gz) =
>>
>> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
>> 1c1
>> < SHA512 (hadoop-2.10.2-src.tar.gz) =
>>
>> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
>> ---
>> > SHA512 (target/artifacts/hadoop-2.10.2-src.tar.gz) =
>>
>> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
>> 1c1
>> < SHA512 (hadoop-2.10.2.tar.gz) =
>>
>> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
>> ---
>> > SHA512 (target/artifacts/hadoop-2.10.2.tar.gz) =
>>
>> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
>>
>> However, checksums are correct.
>>
>> * Builds from source look good
>>  - mvn clean install  -DskipTests
>>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Rat check, if run before building from source locally, fails with error:
>>
>> [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:2.10.2 or one of its
>> dependencies could not be resolved: Could not find artifact
>> org.apache.hadoop:hadoop-maven-plugins:jar:2.10.2 in central (
>> https://repo.maven.apache.org/maven2) -> [Help 1]
>> [ERROR]
>>
>> However, once we build locally, rat check passes (because
>> hadoop-maven-plugins 2.10.2 would be present in local .m2).
>> Also, hadoop-maven-plugins:2.10.2 is available here
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1350/org/apache/hadoop/hadoop-maven-plugins/2.10.2/
>>
>> * Ran sample HDFS and MapReduce commands, look good.
>>
>> Until we release Hadoop artifacts, hadoop-maven-plugins for that release
>> would not be present in the central maven repository, hence I am still
>> wondering how rat check failed only for this RC and not for any of
>> previous
>> release RCs. hadoop-vote.sh always runs rat check before building from
>> source locally.
>>
>>
>> On Tue, May 24, 2022 at 7:41 PM Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp> wrote:
>>
>> > Hi all,
>> >
>> > Here's Hadoop 2.10.2 release candidate #0:
>> >
>> > The RC is available at:
>> >https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
>> >
>> > The RC tag is at:
>> >https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
>> >
>> > The Maven artifacts are staged at:
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1350
>> >
>> > You can find my public key at:
>> >https://downloads.apache.org/hadoop/common/KEYS
>> >
>> > Please evaluate the RC and vote.
>> > The vote will be open for (at least) 5 days.
>> >
>> > Thanks,
>> > Masatake Iwasaki
>> >
>> > -
>> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Viraj Jasani]

+0 (non-binding),

* Signature/Checksum looks good, though I am not sure where
"target/artifacts" is coming from for the tars, here is the diff (this was
the case for 2.10.1 as well but checksum was correct):

1c1
< SHA512 (hadoop-2.10.2-site.tar.gz) =
3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
---
> SHA512 (target/artifacts/hadoop-2.10.2-site.tar.gz) =
3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
1c1
< SHA512 (hadoop-2.10.2-src.tar.gz) =
483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
---
> SHA512 (target/artifacts/hadoop-2.10.2-src.tar.gz) =
483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
1c1
< SHA512 (hadoop-2.10.2.tar.gz) =
13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
---
> SHA512 (target/artifacts/hadoop-2.10.2.tar.gz) =
13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576

However, checksums are correct.

* Builds from source look good
 - mvn clean install  -DskipTests
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Rat check, if run before building from source locally, fails with error:

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:2.10.2 or one of its
dependencies could not be resolved: Could not find artifact
org.apache.hadoop:hadoop-maven-plugins:jar:2.10.2 in central (
https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR]

However, once we build locally, rat check passes (because
hadoop-maven-plugins 2.10.2 would be present in local .m2).
Also, hadoop-maven-plugins:2.10.2 is available here
https://repository.apache.org/content/repositories/orgapachehadoop-1350/org/apache/hadoop/hadoop-maven-plugins/2.10.2/

* Ran sample HDFS and MapReduce commands, look good.

Until we release Hadoop artifacts, hadoop-maven-plugins for that release
would not be present in the central maven repository, hence I am still
wondering how rat check failed only for this RC and not for any of previous
release RCs. hadoop-vote.sh always runs rat check before building from
source locally.


On Tue, May 24, 2022 at 7:41 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Hi all,
>
> Here's Hadoop 2.10.2 release candidate #0:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1350
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Viraj Jasani]

+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS, MapReduce and HBase (2.5) CRUD functional testing on
pseudo-distributed mode looks good.


On Wed, May 11, 2022 at 10:26 AM Steve Loughran 
wrote:

> I have put together a release candidate (RC1) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/
>
> The git tag is release-3.3.3-RC1, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1349/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
> * The critical fixes which shipped in the 3.2.3 release.
> * CVEs in our code and dependencies
> * Shaded client packaging issues.
> * A switch from log4j to reload4j
>
> reload4j is an active fork of the log4j 1.17 library with the classes
> which contain CVEs removed. Even though hadoop never used those classes,
> they regularly raised alerts on security scans and concen from users.
> Switching to the forked project allows us to ship a secure logging
> framework. It will complicate the builds of downstream
> maven/ivy/gradle projects which exclude our log4j artifacts, as they
> need to cut the new dependency instead/as well.
>
> See the release notes for details.
>
> This is the second release attempt. It is the same git commit as before,
> but
> fully recompiled with another republish to maven staging, which has bee
> verified by building spark, as well as a minimal test project.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.3

[Viraj Jasani]

+1 (non-binding),

With a minor change  in
hadoop-vote,

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

HDFS and MapReduce functional testing looks good.

As per PR#4268 , except for a
few flakes, TestDistributedShell and TestCsiClient are consistently failing.


On Tue, May 3, 2022 at 4:24 AM Steve Loughran 
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>- The critical fixes which shipped in the 3.2.3 release.
>-  CVEs in our code and dependencies
>- Shaded client packaging issues.
>- A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing &c to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

[jira] [Created] (HADOOP-18228) Update hadoop-vote to use HADOOP_RC_VERSION dir

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18228:
-

 Summary: Update hadoop-vote to use HADOOP_RC_VERSION dir
 Key: HADOOP-18228
 URL: https://issues.apache.org/jira/browse/HADOOP-18228
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


The recent changes in release script requires a minor change in hadoop-vote to 
use Hadoop RC version dir before verifying signature and checksum of .tar.gz 
files.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18224) Upgrade maven compiler plugin

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18224:
-

 Summary: Upgrade maven compiler plugin
 Key: HADOOP-18224
 URL: https://issues.apache.org/jira/browse/HADOOP-18224
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Currently we are using maven-compiler-plugin 3.1 version, which is quite old 
(2013) and it's also pulling in vulnerable log4j dependency:
{code:java}
[INFO]
org.apache.maven.plugins:maven-compiler-plugin:maven-plugin:3.1:runtime
[INFO]   org.apache.maven.plugins:maven-compiler-plugin:jar:3.1
[INFO]   org.apache.maven:maven-plugin-api:jar:2.0.9
[INFO]   org.apache.maven:maven-artifact:jar:2.0.9
[INFO]   org.codehaus.plexus:plexus-utils:jar:1.5.1
[INFO]   org.apache.maven:maven-core:jar:2.0.9
[INFO]   org.apache.maven:maven-settings:jar:2.0.9
[INFO]   org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.9
...
...
...
[INFO]   log4j:log4j:jar:1.2.12
[INFO]   commons-logging:commons-logging-api:jar:1.1
[INFO]   com.google.collections:google-collections:jar:1.0
[INFO]   junit:junit:jar:3.8.2
 {code}
 

We should upgrade to 3.10.1 (latest Mar, 2022) version of maven-compiler-plugin.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18196) Remove replace-guava from replacer plugin

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18196:
-

 Summary: Remove replace-guava from replacer plugin
 Key: HADOOP-18196
 URL: https://issues.apache.org/jira/browse/HADOOP-18196
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani


While running the build, realized that all replacer plugin executions run only 
after "banned-illegal-imports" enforcer plugin.

For instance,
{code:java}
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (banned-illegal-imports) @ 
hadoop-cloud-storage ---
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-generated-sources) @ 
hadoop-cloud-storage ---
[INFO] Skipping
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-sources) @ hadoop-cloud-storage ---
[INFO] Skipping
[INFO] 
[INFO] --- replacer:1.5.3:replace (replace-guava) @ hadoop-cloud-storage ---
[INFO] Replacement run on 0 file.
[INFO]  {code}
Hence, if our source code uses com.google.common, banned-illegal-imports will 
cause the build failure and replacer plugin would not even get executed.

We should remove it as it is only redundant execution step.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18191) Log retry count while handling exceptions in RetryInvocationHandler

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18191:
-

 Summary: Log retry count while handling exceptions in 
RetryInvocationHandler
 Key: HADOOP-18191
 URL: https://issues.apache.org/jira/browse/HADOOP-18191
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As part of failure handling in RetryInvocationHandler, we log details of the 
Exception details with which API was invoked, failover attempts, delay.

For the purpose of better debugging as well as fine-tuning of retry params, it 
would be good to also log retry count that we already maintain in the Counter 
object.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC1

[Viraj Jasani]

+1 (non-binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Functional testing of HDFS with HBase looks good (from RC0)
* MapReduce functional testing looks good
* Re-run of UTs for all modules looks good


On Sun, Mar 20, 2022 at 11:03 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Hi all,
>
> Here's Hadoop 3.2.3 release candidate #1:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-3.2.3-RC1/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC1
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1342
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

[Viraj Jasani]

+1 (non-binding)

Using hadoop-vote.sh

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Tested HDFS on pseudo-distributed mode with HBase 2.4 pseudo-distributed
cluster (1M rows ingested), all good.

Test PR to run full build and track UT failures
https://github.com/apache/hadoop/pull/4073, few tests are flaky but they
are passing locally.


On Mon, Mar 14, 2022 at 12:45 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Hi all,
>
> Here's Hadoop 3.2.3 release candidate #0:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-3.2.3-RC0/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC0
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1339
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
>
> Thanks,
> Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC5

[Viraj Jasani]

+1 (non-binding)

Using hadoop-vote.sh:

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true


* Basic functional testing on pseudo distributed cluster (carry-forwarded
from RC4): HDFS, MapReduce, ATSv2, HBase (2.x)
* Jira fixVersions seem consistent with git commits



On Tue, Feb 22, 2022 at 10:47 AM Chao Sun  wrote:

> Hi all,
>
> Here's Hadoop 3.3.2 release candidate #5:
>
> The RC is available at: http://people.apache.org/~sunchao/hadoop-3.3.2-RC5
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC5
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1335
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> CHANGELOG is the only difference between this and RC4. Therefore, the tests
> I've done in RC4 are still valid:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested a few simple commands
> - Ran all the tests in Spark using the RC5 artifacts
>
> Please evaluate the RC and vote, thanks!
>
> Best,
> Chao
>

[jira] [Created] (HADOOP-18142) Increase precommit job timeout from 24 hr to 30 hr

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18142:
-

 Summary: Increase precommit job timeout from 24 hr to 30 hr
 Key: HADOOP-18142
 URL: https://issues.apache.org/jira/browse/HADOOP-18142
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As per some recent precommit build results, full build QA is not getting 
completed in 24 hr (recent example 
[here|https://github.com/apache/hadoop/pull/4000] where more than 5 builds 
timed out after 24 hr). We should increase it to 30 hr.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18131) Upgrade maven enforcer plugin and relevant dependencies

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18131:
-

 Summary: Upgrade maven enforcer plugin and relevant dependencies
 Key: HADOOP-18131
 URL: https://issues.apache.org/jira/browse/HADOOP-18131
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Maven enforcer plugin's latest version 3.0.0 has some noticeable improvements 
(e.g. MENFORCER-350, MENFORCER-388, MENFORCER-353) and fixes for us to 
incorporate. Besides, some of the relevant enforcer dependencies (e.g. extra 
enforcer rules and restrict import enforcer) too have good improvements.

We should upgrade maven enforcer plugin and the relevant dependencies.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC4

[Viraj Jasani]

Thanks Chao, things look good with the Jira fixVersion discrepancies.
I am curious if this requires new RC or you would like to update Changelog
in the existing RC. Either way, once Changelog is updated, and since other
functional testing looks good anyways, my vote would be +1 (non-binding).


On Tue, 15 Feb 2022 at 11:27 PM, Chao Sun  wrote:

> Thanks again Viraj!
>
> > 1. Incorrect fix versions: HADOOP-17988, HADOOP-17728
>
> I fixed HADOOP-17988. HADOOP-17728 is resolved as invalid so I removed the
> fix version tag.
>
> 2. HADOOP-17873 is not yet resolved but the corresponding commit is
> present.
>
> Removed the fix version tag for this too.
>
> > 3. Resolved with 3.3.2 fixversion but no no corresponding commit found
> on 3.3.2: HADOOP-17936, HADOOP-18066
>
> I think HADOOP-17936 is already in branch-3.3.2?
> https://github.com/apache/hadoop/commit/6931b70a004de2c84b0630bdb92a2c9f62879c24
> HADOOP-18066 is resolved as invalid so I just removed the fix version tag.
>
> Please check again when you have a chance. Thanks.
>
> Best,
> Chao
>
>
>
>
>
>
>
>
>
>
> On Mon, Feb 14, 2022 at 11:51 PM Viraj Jasani  wrote:
>
>> Thanks Chao. I just re-executed the script and found very minor
>> differences:
>>
>> 1. Incorrect fix versions:
>> HADOOP-17988
>> HADOOP-17728
>>
>> 2. HADOOP-17873 is not yet resolved but the corresponding commit is
>> present.
>>
>> 3. Resolved with 3.3.2 fixversion but no no corresponding commit found on
>> 3.3.2:
>> HADOOP-17936
>> HADOOP-18066
>>
>>
>>
>> On Mon, Feb 14, 2022 at 11:49 PM Chao Sun  wrote:
>>
>>> Thank you so much Viraj! I've no idea that I missed tagging these many
>>> JIRAs. The script is super useful!
>>>
>>> I just fixed the "fix version" of these. Could you double check? Really
>>> appreciate you putting effort to do a thorough check on this.
>>>
>>> Chao
>>>
>>> On Mon, Feb 14, 2022 at 3:10 AM Viraj Jasani  wrote:
>>>
>>>> -0 (non-binding), due to git/jira version discrepancies. Once resolved
>>>> and
>>>> changelist updated, will change my vote to +1 (non-binding).
>>>>
>>>> Basic RC verification using hadoop-vote.sh
>>>> <https://github.com/apache/hadoop/blob/trunk/dev-support/hadoop-vote.sh
>>>> >:
>>>>
>>>> * Signature: ok
>>>> * Checksum : ok
>>>> * Rat check (1.8.0_301): ok
>>>>  - mvn clean apache-rat:check
>>>> * Built from source (1.8.0_301): ok
>>>>  - mvn clean install  -DskipTests
>>>> * Built tar from source (1.8.0_301): ok
>>>>  - mvn clean package  -Pdist -DskipTests -Dtar
>>>> -Dmaven.javadoc.skip=true
>>>>
>>>> Functional validation of HDFS, MapReduce and ATSv2 carry-forwarded from
>>>> previous RC.
>>>>
>>>> Found some Git/Jira version discrepancies (script used from PR
>>>> <https://github.com/apache/hadoop/pull/3991>):
>>>>
>>>> Jiras with missing fixVersion: 3.3.2
>>>>
>>>> HADOOP-17198
>>>> HDFS-16344
>>>> HDFS-16339
>>>> YARN-11007
>>>> HDFS-16171
>>>> HDFS-16350
>>>> HDFS-16336
>>>> HADOOP-17975
>>>> YARN-10991
>>>> HDFS-16332
>>>> HADOOP-17857
>>>> HDFS-16271
>>>> HADOOP-17195
>>>> HADOOP-17290
>>>> HADOOP-17819
>>>> YARN-9551
>>>>
>>>>
>>>>
>>>> On Mon, Feb 14, 2022 at 11:10 AM Viraj Jasani 
>>>> wrote:
>>>>
>>>> > Thanks Chao. Yes contents are now present, I will start the testing.
>>>> >
>>>> >
>>>> > On Mon, 14 Feb 2022 at 2:13 AM, Chao Sun  wrote:
>>>> >
>>>> >> Oops, sorry about that. They should be there now. Can you check
>>>> again?
>>>> >> Thanks!
>>>> >>
>>>> >> Chao
>>>> >>
>>>> >> On Sun, Feb 13, 2022 at 6:47 AM Viraj Jasani 
>>>> wrote:
>>>> >>
>>>> >>> > The RC is available at:
>>>> >>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>>>> >>>
>>>> >>> Chao, the RC folder seems empty as of now.
>>>> >>>
>>>> >>&g

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC4

[Viraj Jasani]

Thanks Chao. I just re-executed the script and found very minor differences:

1. Incorrect fix versions:
HADOOP-17988
HADOOP-17728

2. HADOOP-17873 is not yet resolved but the corresponding commit is present.

3. Resolved with 3.3.2 fixversion but no no corresponding commit found on
3.3.2:
HADOOP-17936
HADOOP-18066



On Mon, Feb 14, 2022 at 11:49 PM Chao Sun  wrote:

> Thank you so much Viraj! I've no idea that I missed tagging these many
> JIRAs. The script is super useful!
>
> I just fixed the "fix version" of these. Could you double check? Really
> appreciate you putting effort to do a thorough check on this.
>
> Chao
>
> On Mon, Feb 14, 2022 at 3:10 AM Viraj Jasani  wrote:
>
>> -0 (non-binding), due to git/jira version discrepancies. Once resolved and
>> changelist updated, will change my vote to +1 (non-binding).
>>
>> Basic RC verification using hadoop-vote.sh
>> <https://github.com/apache/hadoop/blob/trunk/dev-support/hadoop-vote.sh>:
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_301): ok
>>  - mvn clean apache-rat:check
>> * Built from source (1.8.0_301): ok
>>  - mvn clean install  -DskipTests
>> * Built tar from source (1.8.0_301): ok
>>  - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>>
>> Functional validation of HDFS, MapReduce and ATSv2 carry-forwarded from
>> previous RC.
>>
>> Found some Git/Jira version discrepancies (script used from PR
>> <https://github.com/apache/hadoop/pull/3991>):
>>
>> Jiras with missing fixVersion: 3.3.2
>>
>> HADOOP-17198
>> HDFS-16344
>> HDFS-16339
>> YARN-11007
>> HDFS-16171
>> HDFS-16350
>> HDFS-16336
>> HADOOP-17975
>> YARN-10991
>> HDFS-16332
>> HADOOP-17857
>> HDFS-16271
>> HADOOP-17195
>> HADOOP-17290
>> HADOOP-17819
>> YARN-9551
>>
>>
>>
>> On Mon, Feb 14, 2022 at 11:10 AM Viraj Jasani  wrote:
>>
>> > Thanks Chao. Yes contents are now present, I will start the testing.
>> >
>> >
>> > On Mon, 14 Feb 2022 at 2:13 AM, Chao Sun  wrote:
>> >
>> >> Oops, sorry about that. They should be there now. Can you check again?
>> >> Thanks!
>> >>
>> >> Chao
>> >>
>> >> On Sun, Feb 13, 2022 at 6:47 AM Viraj Jasani 
>> wrote:
>> >>
>> >>> > The RC is available at:
>> >>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>> >>>
>> >>> Chao, the RC folder seems empty as of now.
>> >>>
>> >>>
>> >>> On Sat, Feb 12, 2022 at 3:12 AM Chao Sun  wrote:
>> >>>
>> >>>> Hi all,
>> >>>>
>> >>>> Sorry for the delay! was waiting for a few fixes that people have
>> >>>> requested
>> >>>> to backport. I've just put together Hadoop 3.3.2 RC4 below:
>> >>>>
>> >>>> The RC is available at:
>> >>>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>> >>>> The RC tag is at:
>> >>>> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC4
>> >>>> The Maven artifacts are staged at:
>> >>>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1334/
>> >>>>
>> >>>> You can find my public key at:
>> >>>> https://downloads.apache.org/hadoop/common/KEYS
>> >>>>
>> >>>> The deltas between this and RC3 are the addition of the following
>> fixes:
>> >>>>   - HADOOP-17198. Support S3 Access Points
>> >>>>   - HADOOP-18096. Distcp: Sync moves filtered file to home directory
>> >>>> rather
>> >>>> than deleting.
>> >>>>   - YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN
>> >>>> application catalog webapp
>> >>>>
>> >>>> Same as before, I've done the following verification and they look
>> good:
>> >>>> - Ran all the unit tests
>> >>>> - Started a single node HDFS cluster and tested a few simple commands
>> >>>> - Ran all the tests in Spark using the RC4 artifacts
>> >>>>
>> >>>> Please evaluate the RC and vote, thanks!
>> >>>>
>> >>>> Best,
>> >>>> Chao
>> >>>>
>> >>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC4

[Viraj Jasani]

-0 (non-binding), due to git/jira version discrepancies. Once resolved and
changelist updated, will change my vote to +1 (non-binding).

Basic RC verification using hadoop-vote.sh
<https://github.com/apache/hadoop/blob/trunk/dev-support/hadoop-vote.sh>:

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
 - mvn clean package  -Pdist -DskipTests -Dtar
-Dmaven.javadoc.skip=true

Functional validation of HDFS, MapReduce and ATSv2 carry-forwarded from
previous RC.

Found some Git/Jira version discrepancies (script used from PR
<https://github.com/apache/hadoop/pull/3991>):

Jiras with missing fixVersion: 3.3.2

HADOOP-17198
HDFS-16344
HDFS-16339
YARN-11007
HDFS-16171
HDFS-16350
HDFS-16336
HADOOP-17975
YARN-10991
HDFS-16332
HADOOP-17857
HDFS-16271
HADOOP-17195
HADOOP-17290
HADOOP-17819
YARN-9551



On Mon, Feb 14, 2022 at 11:10 AM Viraj Jasani  wrote:

> Thanks Chao. Yes contents are now present, I will start the testing.
>
>
> On Mon, 14 Feb 2022 at 2:13 AM, Chao Sun  wrote:
>
>> Oops, sorry about that. They should be there now. Can you check again?
>> Thanks!
>>
>> Chao
>>
>> On Sun, Feb 13, 2022 at 6:47 AM Viraj Jasani  wrote:
>>
>>> > The RC is available at:
>>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>>>
>>> Chao, the RC folder seems empty as of now.
>>>
>>>
>>> On Sat, Feb 12, 2022 at 3:12 AM Chao Sun  wrote:
>>>
>>>> Hi all,
>>>>
>>>> Sorry for the delay! was waiting for a few fixes that people have
>>>> requested
>>>> to backport. I've just put together Hadoop 3.3.2 RC4 below:
>>>>
>>>> The RC is available at:
>>>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>>>> The RC tag is at:
>>>> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC4
>>>> The Maven artifacts are staged at:
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1334/
>>>>
>>>> You can find my public key at:
>>>> https://downloads.apache.org/hadoop/common/KEYS
>>>>
>>>> The deltas between this and RC3 are the addition of the following fixes:
>>>>   - HADOOP-17198. Support S3 Access Points
>>>>   - HADOOP-18096. Distcp: Sync moves filtered file to home directory
>>>> rather
>>>> than deleting.
>>>>   - YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN
>>>> application catalog webapp
>>>>
>>>> Same as before, I've done the following verification and they look good:
>>>> - Ran all the unit tests
>>>> - Started a single node HDFS cluster and tested a few simple commands
>>>> - Ran all the tests in Spark using the RC4 artifacts
>>>>
>>>> Please evaluate the RC and vote, thanks!
>>>>
>>>> Best,
>>>> Chao
>>>>
>>>

[jira] [Created] (HADOOP-18125) Utility to identify git commit / Jira fixVersion discrepancies for RC preparation

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18125:
-

 Summary: Utility to identify git commit / Jira fixVersion 
discrepancies for RC preparation
 Key: HADOOP-18125
 URL: https://issues.apache.org/jira/browse/HADOOP-18125
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As part of RC preparation,  we need to identify all git commits that landed on 
release branch, however their corresponding Jira is either not resolved yet or 
does not contain expected fixVersions. Only when we have git commits and 
corresponding Jiras with expected fixVersion resolved, we get all such Jiras 
included in auto-generated CHANGES.md as per Yetus changelog generator.

Proposal of this Jira is to provide such script that can be useful for all 
upcoming RC preparations and list down all Jiras where we need manual 
intervention. This utility script should use Jira API to retrieve individual 
fields and use git log to loop through commit history.

The script should identify these issues:
 # commit is reverted as per commit message
 # commit does not contain Jira number format (e.g. HADOOP- / HDFS- 
etc) in message
 # Jira does not have expected fixVersion
 # Jira has expected fixVersion, but it is not yet resolved
 # Jira has release corresponding fixVersion and is resolved, but no 
corresponding commit yet found

It can take inputs as:
 # First commit hash to start excluding commits from history
 # Fix Version
 # JIRA Project Name
 # Path of project's working dir
 # Jira server url



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC4

[Viraj Jasani]

Thanks Chao. Yes contents are now present, I will start the testing.


On Mon, 14 Feb 2022 at 2:13 AM, Chao Sun  wrote:

> Oops, sorry about that. They should be there now. Can you check again?
> Thanks!
>
> Chao
>
> On Sun, Feb 13, 2022 at 6:47 AM Viraj Jasani  wrote:
>
>> > The RC is available at:
>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>>
>> Chao, the RC folder seems empty as of now.
>>
>>
>> On Sat, Feb 12, 2022 at 3:12 AM Chao Sun  wrote:
>>
>>> Hi all,
>>>
>>> Sorry for the delay! was waiting for a few fixes that people have
>>> requested
>>> to backport. I've just put together Hadoop 3.3.2 RC4 below:
>>>
>>> The RC is available at:
>>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
>>> The RC tag is at:
>>> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC4
>>> The Maven artifacts are staged at:
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1334/
>>>
>>> You can find my public key at:
>>> https://downloads.apache.org/hadoop/common/KEYS
>>>
>>> The deltas between this and RC3 are the addition of the following fixes:
>>>   - HADOOP-17198. Support S3 Access Points
>>>   - HADOOP-18096. Distcp: Sync moves filtered file to home directory
>>> rather
>>> than deleting.
>>>   - YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN
>>> application catalog webapp
>>>
>>> Same as before, I've done the following verification and they look good:
>>> - Ran all the unit tests
>>> - Started a single node HDFS cluster and tested a few simple commands
>>> - Ran all the tests in Spark using the RC4 artifacts
>>>
>>> Please evaluate the RC and vote, thanks!
>>>
>>> Best,
>>> Chao
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC4

[Viraj Jasani]

> The RC is available at:
http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/

Chao, the RC folder seems empty as of now.


On Sat, Feb 12, 2022 at 3:12 AM Chao Sun  wrote:

> Hi all,
>
> Sorry for the delay! was waiting for a few fixes that people have requested
> to backport. I've just put together Hadoop 3.3.2 RC4 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC4/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC4
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1334/
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> The deltas between this and RC3 are the addition of the following fixes:
>   - HADOOP-17198. Support S3 Access Points
>   - HADOOP-18096. Distcp: Sync moves filtered file to home directory rather
> than deleting.
>   - YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN
> application catalog webapp
>
> Same as before, I've done the following verification and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested a few simple commands
> - Ran all the tests in Spark using the RC4 artifacts
>
> Please evaluate the RC and vote, thanks!
>
> Best,
> Chao
>

[jira] [Created] (HADOOP-18098) Basic verification of release candidates

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18098:
-

 Summary: Basic verification of release candidates
 Key: HADOOP-18098
 URL: https://issues.apache.org/jira/browse/HADOOP-18098
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


We should provide script for the basic sanity of Hadoop release candidates. It 
should include:
 * Signature
 * Checksum
 * Rat check
 * Build from src
 * Build tarball from src

 

Although we can include unit test as well, but overall unit test run is going 
to be significantly higher, and precommit Jenkins builds provide better view of 
UT sanity.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

[Viraj Jasani]

> * ATSv2 functional tests: ok

Ran against hbase 1.7 cluster

On Thu, Jan 27, 2022 at 5:03 PM Viraj Jasani  wrote:

> +1 (non-binding)
>
> * Signature: ok
> * Checksum: ok
> * Build from source: ok
> * Ran some large tests from hbase 2.5 branch against RC2: looks good
> (carry forward from previous RC)
> * HDFS functional tests: ok
> * Ran a couple of MapReduce Jobs: ok
> * ATSv2 functional tests: ok
>
>
> On Thu, Jan 27, 2022 at 12:47 AM Chao Sun  wrote:
>
>> Hi all,
>>
>> I've put together Hadoop 3.3.2 RC3 below:
>>
>> The RC is available at:
>> http://people.apache.org/~sunchao/hadoop-3.3.2-RC3/
>> The RC tag is at:
>> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC3
>> The Maven artifacts are staged at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1333
>>
>> You can find my public key at:
>> https://downloads.apache.org/hadoop/common/KEYS
>>
>> The only delta between this and RC2 is the addition of the following fix:
>>   - HADOOP-18094. Disable S3A auditing by default.
>>
>> I've done the same tests as in RC2 and they look good:
>> - Ran all the unit tests
>> - Started a single node HDFS cluster and tested a few simple commands
>> - Ran all the tests in Spark using the RC2 artifacts
>>
>> Please evaluate the RC and vote, thanks!
>>
>> Best,
>> Chao
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

[Viraj Jasani]

+1 (non-binding)

* Signature: ok
* Checksum: ok
* Build from source: ok
* Ran some large tests from hbase 2.5 branch against RC2: looks good (carry
forward from previous RC)
* HDFS functional tests: ok
* Ran a couple of MapReduce Jobs: ok
* ATSv2 functional tests: ok


On Thu, Jan 27, 2022 at 12:47 AM Chao Sun  wrote:

> Hi all,
>
> I've put together Hadoop 3.3.2 RC3 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC3/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC3
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1333
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> The only delta between this and RC2 is the addition of the following fix:
>   - HADOOP-18094. Disable S3A auditing by default.
>
> I've done the same tests as in RC2 and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested a few simple commands
> - Ran all the tests in Spark using the RC2 artifacts
>
> Please evaluate the RC and vote, thanks!
>
> Best,
> Chao
>

[jira] [Created] (HADOOP-18089) Test coverage for Async profiler servlets

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18089:
-

 Summary: Test coverage for Async profiler servlets
 Key: HADOOP-18089
 URL: https://issues.apache.org/jira/browse/HADOOP-18089
 Project: Hadoop Common
  Issue Type: Test
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As discussed in HADOOP-18077, we should provide sufficient test coverage to 
discover any potential regression in async profiler servlets: ProfileServlet 
and ProfileOutputServlet.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18077) ProfileOutputServlet unable to proceed due to NPE

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18077:
-

 Summary: ProfileOutputServlet unable to proceed due to NPE
 Key: HADOOP-18077
 URL: https://issues.apache.org/jira/browse/HADOOP-18077
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


ProfileOutputServlet context doesn't have Hadoop configs available and hence 
async profiler redirection to output servlet is failing to identify if admin 
access is allowed:
{code:java}
HTTP ERROR 500 java.lang.NullPointerException
URI:    /prof-output-hadoop/async-prof-pid-98613-cpu-2.html
STATUS:    500
MESSAGE:    java.lang.NullPointerException
SERVLET:    org.apache.hadoop.http.ProfileOutputServlet-58c34bb3
CAUSED BY:    java.lang.NullPointerException
Caused by:
java.lang.NullPointerException
    at 
org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1619)
    at 
org.apache.hadoop.http.ProfileOutputServlet.doGet(ProfileOutputServlet.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
    at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    at 
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
    at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18055) Async Profiler endpoint for Hadoop daemons

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18055:
-

 Summary: Async Profiler endpoint for Hadoop daemons
 Key: HADOOP-18055
 URL: https://issues.apache.org/jira/browse/HADOOP-18055
 Project: Hadoop Common
  Issue Type: New Feature
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Async profiler ([https://github.com/jvm-profiling-tools/async-profiler]) is a 
low overhead sampling profiler for Java that does not suffer from Safepoint 
bias problem. It features HotSpot-specific APIs to collect stack traces and to 
track memory allocations. The profiler works with OpenJDK, Oracle JDK and other 
Java runtimes based on the HotSpot JVM.

Async profiler can also profile heap allocations, lock contention, and HW 
performance counters in addition to CPU.

We have an httpserver based servlet stack hence we can use HIVE-20202 as an 
implementation template to provide async profiler as servlet for Hadoop 
daemons. Ideally we achieve these requirements:
 * Retrieve flamegraph SVG generated from latest profile trace.
 * Online enable and disable of profiling activity. (async-profiler does not do 
instrumentation based profiling so this should not cause the code gen related 
perf problems of that other approach and can be safely toggled on and off while 
under production load.)
 * CPU profiling.
 * ALLOCATION profiling.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18039) Upgrade hbase2 version and fix TestTimelineWriterHBaseDown

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18039:
-

 Summary: Upgrade hbase2 version and fix TestTimelineWriterHBaseDown
 Key: HADOOP-18039
 URL: https://issues.apache.org/jira/browse/HADOOP-18039
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As mentioned on the parent Jira, we can't upgrade hbase2 profile version beyond 
2.2.4 until we either have hbase 2 artifacts available that are built with 
hadoop 3 profile by default or hbase 3 is rolled out (hbase 3 is compatible 
with hadoop 3 versions only).

Let's upgrade hbase2 profile version to 2.2.4 as part of this Jira and also fix 
TestTimelineWriterHBaseDown to create connection only after mini cluster is up.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18027) Include static imports in the maven plugin rules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18027:
-

 Summary: Include static imports in the maven plugin rules
 Key: HADOOP-18027
 URL: https://issues.apache.org/jira/browse/HADOOP-18027
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Maven enforcer plugin to ban illegal imports require explicit mention of static 
imports in order to evaluate whether any publicly accessible static entities 
from the banned classes are directly imported by Hadoop code.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18025) Upgrade HBase version to 1.7.1 for hbase1 profile

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18025:
-

 Summary: Upgrade HBase version to 1.7.1 for hbase1 profile
 Key: HADOOP-18025
 URL: https://issues.apache.org/jira/browse/HADOOP-18025
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: yetus isn't working

[Viraj Jasani]

This does seem to be happening intermittently. For one of the recent
builds, 4/5 runs failed with the same reason and only 1/5 was able to
proceed further.

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3711/


On Wed, 24 Nov 2021 at 11:28 PM, Ayush Saxena  wrote:

> This was happening with this PR only, others seems to work.
>
> I was trying to fix, by then you merged it and the build got disabled.
>
> Took me 10 mins to realise that the PR got merged, that is why the build
> got disabled and it is not me who broke it in experimenting. :-)
>
> I will try to fix it next time….
>
>
> On 24-Nov-2021, at 10:36 PM, Steve Loughran 
> wrote:
> >
> > 
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3633/9/pipeline
> >
> > Check out from version control
> > <1s
> > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3633:
> Read-only
> > file system
> >
> > #!/usr/bin/env bash # See HADOOP-13951 chmod -R u+rxw "${WORKSPACE}"
> > — Shell Script
> > <1s
> > Required context class hudson.FilePath is missing
> >
> > Perhaps you forgot to surround the code with a step that provides this,
> > such as: node
> >
> > anyone know how to fix this?
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-18022) Add restrict-imports-enforcer-rule for Guava Preconditions in hadoop-main pom

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18022:
-

 Summary: Add restrict-imports-enforcer-rule for Guava 
Preconditions in hadoop-main pom
 Key: HADOOP-18022
 URL: https://issues.apache.org/jira/browse/HADOOP-18022
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Add restrict-imports-enforcer-rule for Guava Preconditions in hadoop-main pom 
to restrict any new import in future. Remove any remaining usages of Guava 
Preconditions from the codebase.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18018) unguava: remove Preconditions from hadoop-tools modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18018:
-

 Summary: unguava: remove Preconditions from hadoop-tools modules
 Key: HADOOP-18018
 URL: https://issues.apache.org/jira/browse/HADOOP-18018
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Replace guava Preconditions by internal implementations that rely on java8+ 
APIs in the hadoop.util for all modules in hadoop-tools.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18017) unguava: remove Preconditions from hadoop-yarn-project modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18017:
-

 Summary: unguava: remove Preconditions from hadoop-yarn-project 
modules
 Key: HADOOP-18017
 URL: https://issues.apache.org/jira/browse/HADOOP-18017
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Replace guava Preconditions by internal implementations that rely on java8+ 
APIs in the hadoop.util for all modules in hadoop-yarn-project.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18006) maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-18006:
-

 Summary: maven-enforcer-plugin's execution of 
banned-illegal-imports gets overridden in child poms
 Key: HADOOP-18006
 URL: https://issues.apache.org/jira/browse/HADOOP-18006
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


When we specify any maven plugin with execution tag in the parent as well as 
child modules, child module plugin overrides parent plugin. For instance, when 
{{banned-illegal-imports}} is applied for any child module with only one banned 
import (let’s say {{{}Preconditions{}}}), then only that banned import is 
covered by that child module and all imports defined in parent module (e.g 
Sets, Lists etc) are overridden and they are no longer applied.
After this 
[commit|https://github.com/apache/hadoop/commit/62c86eaa0e539a4307ca794e0fcd502a77ebceb8],
 hadoop-hdfs module will not complain about {{Sets}} even if i import it from 
guava banned imports but on the other hand, hadoop-yarn module doesn’t have any 
child level {{banned-illegal-imports}} defined so yarn modules will fail if 
{{Sets}} guava import is used.
So going forward, it would be good to replace guava imports with Hadoop’s own 
imports module-by-module and only at the end, we should add new entry to parent 
pom {{banned-illegal-imports}} list.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17968) Migrate checkstyle IllegalImport to banned-illegal-imports enforcer

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17968:
-

 Summary: Migrate checkstyle IllegalImport to 
banned-illegal-imports enforcer
 Key: HADOOP-17968
 URL: https://issues.apache.org/jira/browse/HADOOP-17968
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As discussed on PR [3503|https://github.com/apache/hadoop/pull/3503], we should 
migrate existing imports provided in IllegalImport tag in checkstyle.xml to 
maven-enforcer-plugin's banned-illegal-imports enforcer rule so that build 
never succeeds in the presence of any of the illegal imports.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17967) Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17967:
-

 Summary: Keep restrict-imports-enforcer-rule for Guava 
VisibleForTesting in hadoop-main pom
 Key: HADOOP-17967
 URL: https://issues.apache.org/jira/browse/HADOOP-17967
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17963) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17963:
-

 Summary: Replace Guava VisibleForTesting by Hadoop's own 
annotation in hadoop-yarn-project modules
 Key: HADOOP-17963
 URL: https://issues.apache.org/jira/browse/HADOOP-17963
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17962) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-tools modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17962:
-

 Summary: Replace Guava VisibleForTesting by Hadoop's own 
annotation in hadoop-tools modules
 Key: HADOOP-17962
 URL: https://issues.apache.org/jira/browse/HADOOP-17962
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17959) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17959:
-

 Summary: Replace Guava VisibleForTesting by Hadoop's own 
annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules
 Key: HADOOP-17959
 URL: https://issues.apache.org/jira/browse/HADOOP-17959
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17957) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17957:
-

 Summary: Replace Guava VisibleForTesting by Hadoop's own 
annotation in hadoop-hdfs-project modules
 Key: HADOOP-17957
 URL: https://issues.apache.org/jira/browse/HADOOP-17957
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17956) Replace all default Charset usage with UTF-8

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17956:
-

 Summary: Replace all default Charset usage with UTF-8
 Key: HADOOP-17956
 URL: https://issues.apache.org/jira/browse/HADOOP-17956
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


As discussed on PR#3515, creating this sub-task to replace all default charset 
with UTF-8 as default charset has some potential problems (e.g. HADOOP-11379, 
HADOOP-11389).

FYI [~aajisaka]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-17947) Provide alternative to Guava VisibleForTesting

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani reopened HADOOP-17947:
---

Reopening for a minor addendum.

> Provide alternative to Guava VisibleForTesting
> --
>
> Key: HADOOP-17947
> URL: https://issues.apache.org/jira/browse/HADOOP-17947
> Project: Hadoop Common
>  Issue Type: Sub-task
>        Reporter: Viraj Jasani
>    Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> In an attempt to reduce the dependency on Guava, we should remove 
> VisibleForTesting annotation usages as it has very high usage in our 
> codebase. This Jira is to provide Hadoop's own alternative and use it in 
> hadoop-common-project modules.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17952) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-common-project modules

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17952:
-

 Summary: Replace Guava VisibleForTesting by Hadoop's own 
annotation in hadoop-common-project modules
 Key: HADOOP-17952
 URL: https://issues.apache.org/jira/browse/HADOOP-17952
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17950) Provide replacement for deprecated APIs of commons-io IOUtils

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17950:
-

 Summary: Provide replacement for deprecated APIs of commons-io 
IOUtils
 Key: HADOOP-17950
 URL: https://issues.apache.org/jira/browse/HADOOP-17950
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17947) Provide alternative to Guava VisibleForTesting in Hadoop common

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17947:
-

 Summary: Provide alternative to Guava VisibleForTesting in Hadoop 
common
 Key: HADOOP-17947
 URL: https://issues.apache.org/jira/browse/HADOOP-17947
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Viraj Jasani
Assignee: Viraj Jasani


In an attempt to reduce the dependency on Guava, we should remove 
VisibleForTesting annotation usages as it has very high usage in our codebase. 
This Jira is to provide Hadoop's own alternative and use it in 
hadoop-common-project modules.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Migrate to Yetus Interface classification annotations

[Viraj Jasani]

Thanks Masatake for the suggestions. I agree that until Yetus comes to
final conclusion on whether to keep or drop IA/IS annotations for higher
JDK versions (or fix/drop doclet support), we should hold on for now.
Thanks Sean and Akira for providing the context.


On Tue, 28 Sep 2021 at 6:55 PM, Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Thanks, Akira and Viraj.
>
> My understanding is that we have options like
>
> 1. migrate org.apache.yetus:audience-annotations to Java >= 9 then
> migrate Hadoop to the new org.apache.yetus:audience-annotations.
>
> 2. "use the Jigsaw feature to export only @Public elements to other
> projects
> and create javadoc by new --show-packages=exported option instead of
> relying on the annotations." as mentioned by Akira[1].
>
> Both require dropping Java 8 support.
>
> If current org.apache.yetus:audience-annotations(:0.13.0) for Java 8 no
> longer evolves,
> migrating to it in short term is not much useful?
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-15304?focusedCommentId=16418072&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16418072
>
> On 2021/09/28 18:38, Viraj Jasani wrote:
> >> The problem comes from the removal of com.sun.tools.doclets.* packages
> >
> > Agree. Here is the summary
> > <
> https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html
> >
> > of the replacement package *jdk.javadoc.doclet*.
> > Here is the migration guide
> > <
> https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html#migration
> >
> > for the same.
> >
> > On Tue, Sep 28, 2021 at 1:06 PM Akira Ajisaka 
> wrote:
> >
> >> Hi Masatake,
> >>
> >> The problem comes from the removal of com.sun.tools.doclets.* packages
> in
> >> Java 10.
> >> In Apache Hadoop, I removed the doclet support for filtering javadocs
> when
> >> the environment is Java 10 or upper.
> >> https://issues.apache.org/jira/browse/HADOOP-15304
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Tue, Sep 28, 2021 at 10:27 AM Masatake Iwasaki <
> >> iwasak...@oss.nttdata.co.jp> wrote:
> >>
> >>>> In particular, there has been an outstanding problem with doclet
> >> support
> >>> for filtering javadocs by annotation since JDK9 came out.
> >>>
> >>> Could you give me a pointer to relevant Yetus JIRA or ML thread?
> >>>
> >>> On 2021/09/28 1:17, Sean Busbey wrote:
> >>>> I think consolidating on a common library and tooling for defining API
> >>> expectations for Hadoop would be great.
> >>>>
> >>>> Unfortunately, the Apache Yetus community recently started a
> discussion
> >>> around dropping their maintenance of the audience annotations
> codebase[1]
> >>> due to lack of community interest. In particular, there has been an
> >>> outstanding problem with doclet support for filtering javadocs by
> >>> annotation since JDK9 came out.
> >>>>
> >>>> I think that means a necessary first step here would be to determine
> if
> >>> we have contributors willing to show up over in that project to get
> >> things
> >>> into a good state for future JDK adoption.
> >>>>
> >>>>
> >>>>
> >>>> [1]:
> >>>> https://s.apache.org/ybdl6
> >>>> "[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org
> >>>>
> >>>>> On Sep 27, 2021, at 2:46 AM, Viraj Jasani 
> wrote:
> >>>>>
> >>>>> Since the early days, Hadoop has provided Interface classification
> >>>>> annotations to represent the scope and stability for downstream
> >>>>> applications to select Hadoop APIs carefully. After some time, these
> >>>>> annotations (InterfaceAudience and InterfaceStability) have been
> >>> migrated
> >>>>> to Apache Yetus. As of today, with increasing number of Hadoop
> >> ecosystem
> >>>>> applications using (or starting to use) Yetus stability annotations
> >> for
> >>>>> their own downstreamers, we should also consider using IA/IS
> >> annotations
> >>>>> provided by *org.apache.yetus.audience *directly in our 

Re: [DISCUSS] Migrate to Yetus Interface classification annotations

[Viraj Jasani]

> The problem comes from the removal of com.sun.tools.doclets.* packages

Agree. Here is the summary
<https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html>
of the replacement package *jdk.javadoc.doclet*.
Here is the migration guide
<https://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html#migration>
for the same.

On Tue, Sep 28, 2021 at 1:06 PM Akira Ajisaka  wrote:

> Hi Masatake,
>
> The problem comes from the removal of com.sun.tools.doclets.* packages in
> Java 10.
> In Apache Hadoop, I removed the doclet support for filtering javadocs when
> the environment is Java 10 or upper.
> https://issues.apache.org/jira/browse/HADOOP-15304
>
> Thanks,
> Akira
>
> On Tue, Sep 28, 2021 at 10:27 AM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
> > > In particular, there has been an outstanding problem with doclet
> support
> > for filtering javadocs by annotation since JDK9 came out.
> >
> > Could you give me a pointer to relevant Yetus JIRA or ML thread?
> >
> > On 2021/09/28 1:17, Sean Busbey wrote:
> > > I think consolidating on a common library and tooling for defining API
> > expectations for Hadoop would be great.
> > >
> > > Unfortunately, the Apache Yetus community recently started a discussion
> > around dropping their maintenance of the audience annotations codebase[1]
> > due to lack of community interest. In particular, there has been an
> > outstanding problem with doclet support for filtering javadocs by
> > annotation since JDK9 came out.
> > >
> > > I think that means a necessary first step here would be to determine if
> > we have contributors willing to show up over in that project to get
> things
> > into a good state for future JDK adoption.
> > >
> > >
> > >
> > > [1]:
> > > https://s.apache.org/ybdl6
> > > "[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org
> > >
> > >> On Sep 27, 2021, at 2:46 AM, Viraj Jasani  wrote:
> > >>
> > >> Since the early days, Hadoop has provided Interface classification
> > >> annotations to represent the scope and stability for downstream
> > >> applications to select Hadoop APIs carefully. After some time, these
> > >> annotations (InterfaceAudience and InterfaceStability) have been
> > migrated
> > >> to Apache Yetus. As of today, with increasing number of Hadoop
> ecosystem
> > >> applications using (or starting to use) Yetus stability annotations
> for
> > >> their own downstreamers, we should also consider using IA/IS
> annotations
> > >> provided by *org.apache.yetus.audience *directly in our codebase and
> > retire
> > >> our *org.apache.hadoop.classification* package for the better
> > separation of
> > >> concern and single source.
> > >>
> > >> I believe we can go with this migration to maintain compatibility for
> > >> Hadoop downstreamers:
> > >>
> > >>1. In Hadoop trunk (3.4.0+ releases), replace all usages of o.a.h.c
> > >>stability annotations with o.a.y.a annotations.
> > >>2. Deprecate o.a.h.c annotations, and provide deprecation warning
> > that
> > >>we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only
> > source for
> > >>these annotations should be o.a.y.a.
> > >>
> > >> Any thoughts?
> > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

[DISCUSS] Migrate to Yetus Interface classification annotations

[Viraj Jasani]

Since the early days, Hadoop has provided Interface classification
annotations to represent the scope and stability for downstream
applications to select Hadoop APIs carefully. After some time, these
annotations (InterfaceAudience and InterfaceStability) have been migrated
to Apache Yetus. As of today, with increasing number of Hadoop ecosystem
applications using (or starting to use) Yetus stability annotations for
their own downstreamers, we should also consider using IA/IS annotations
provided by *org.apache.yetus.audience *directly in our codebase and retire
our *org.apache.hadoop.classification* package for the better separation of
concern and single source.

I believe we can go with this migration to maintain compatibility for
Hadoop downstreamers:

   1. In Hadoop trunk (3.4.0+ releases), replace all usages of o.a.h.c
   stability annotations with o.a.y.a annotations.
   2. Deprecate o.a.h.c annotations, and provide deprecation warning that
   we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only source for
   these annotations should be o.a.y.a.

Any thoughts?

Re: [DISCUSS] Checkin Hadoop code formatter

[Viraj Jasani]

PR 3387 <https://github.com/apache/hadoop/pull/3387> has been merged and
the Wiki page is also updated for devs to refer to the IDEA code formatter
xml. If any Eclipse user would like to contribute the formatter xml, please
feel free to comment on HADOOP-17892
<https://issues.apache.org/jira/browse/HADOOP-17892> or create a sub-task.

Thanks to everyone who helped with reviews, merging the PR and updating the
Wiki page.

On Tue, Sep 14, 2021 at 5:29 PM Hui Fei  wrote:

> Thanks Viraj.
> It does make sense.
>
> Viraj Jasani  于2021年9月12日周日 上午2:58写道：
>
>> + common-dev@hadoop.apache.org
>>
>> ------ Forwarded message -
>> From: Viraj Jasani 
>> Date: Tue, Sep 7, 2021 at 6:18 PM
>> Subject: Checkin Hadoop code formatter
>> To: common-dev@hadoop.apache.org 
>>
>>
>> It seems some recent new devs are not familiar with the common code
>> formatter that we use for our codebase.
>> While we already have Wiki page [1] for new contributors and it mentions:
>> "Code must be formatted according to Sun's conventions
>> <
>> http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html
>> >"
>> but this Oracle's code conventions page is not being actively maintained
>> (no update has been received after 1999) and hence, I believe we should
>> check-in and maintain code formatter xmls for supported IDEs in our
>> codebase only (under dev-support) for all devs to be able to import it in
>> the respective IDE.
>> Keeping this in mind, I have created this PR 3387
>> <https://github.com/apache/hadoop/pull/3387>. If you could please take a
>> look and if the PR receives sufficient +1s, we might want to update our
>> Wiki page to directly refer to our own codebase for code formatters that
>> we
>> maintain. Thoughts?
>>
>>
>> 1.
>>
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-MakingChanges
>>
>

[DISCUSS] Checkin Hadoop code formatter

[Viraj Jasani]

+ common-dev@hadoop.apache.org

-- Forwarded message -
From: Viraj Jasani 
Date: Tue, Sep 7, 2021 at 6:18 PM
Subject: Checkin Hadoop code formatter
To: common-dev@hadoop.apache.org 


It seems some recent new devs are not familiar with the common code
formatter that we use for our codebase.
While we already have Wiki page [1] for new contributors and it mentions:
"Code must be formatted according to Sun's conventions
<http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html>"
but this Oracle's code conventions page is not being actively maintained
(no update has been received after 1999) and hence, I believe we should
check-in and maintain code formatter xmls for supported IDEs in our
codebase only (under dev-support) for all devs to be able to import it in
the respective IDE.
Keeping this in mind, I have created this PR 3387
<https://github.com/apache/hadoop/pull/3387>. If you could please take a
look and if the PR receives sufficient +1s, we might want to update our
Wiki page to directly refer to our own codebase for code formatters that we
maintain. Thoughts?


1.
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-MakingChanges

Re: [DISCUSS] Hadoop 3.3.2 release?

[Viraj Jasani]

+1 (non-binding) for the release.


On Tue, 7 Sep 2021 at 10:36 PM, Chao Sun  wrote:

> Hi all,
>
> It has been almost 3 months since the 3.3.1 release and branch-3.3 has
> accumulated quite a few commits (118 atm). In particular, Spark community
> recently found an issue which prevents one from using the shaded Hadoop
> client together with certain compression codecs such as lz4 and snappy
> codec. The details are recorded in HADOOP-17891 and SPARK-36669.
>
> Therefore, I'm wondering if anyone is also interested in a 3.3.2 release.
> If there is no objection, I'd like to volunteer myself for the work as
> well.
>
> Best Regards,
> Chao
>

[jira] [Created] (HADOOP-17892) Add Hadoop code formatter in dev-support

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17892:
-

 Summary: Add Hadoop code formatter in dev-support
 Key: HADOOP-17892
 URL: https://issues.apache.org/jira/browse/HADOOP-17892
 Project: Hadoop Common
  Issue Type: Task
Reporter: Viraj Jasani


We should add Hadoop code formatter xml to dev-support specifically for new 
developers to refer to.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17874) ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner

[Viraj Jasani (Jira)]

Viraj Jasani created HADOOP-17874:
-

 Summary: ExceptionsHandler to add terse/suppressed Exceptions in 
thread-safe manner
 Key: HADOOP-17874
 URL: https://issues.apache.org/jira/browse/HADOOP-17874
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani


Even though we have explicit comments stating that we have thread-safe 
replacement of terseExceptions and suppressedExceptions, in reality we don't 
have it. As we can't guarantee only non-concurrent addition of Exceptions at a 
time from any Server implementation, we should make this thread-safe.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org