[jira] [Resolved] (HADOOP-18962) Upgrade kafka to 3.4.0
[ https://issues.apache.org/jira/browse/HADOOP-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18962. - Fix Version/s: 3.5.0 Resolution: Fixed > Upgrade kafka to 3.4.0 > -- > > Key: HADOOP-18962 > URL: https://issues.apache.org/jira/browse/HADOOP-18962 > Project: Hadoop Common > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Upgrade kafka-clients to 3.4.0 to fix > https://nvd.nist.gov/vuln/detail/CVE-2023-25194 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19168) Upgrade Kafka Clients due to CVEs
[
https://issues.apache.org/jira/browse/HADOOP-19168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19168.
-
Resolution: Duplicate
rohit, dupe of HADOOP-18962. let's focus on that
> Upgrade Kafka Clients due to CVEs
> -
>
> Key: HADOOP-19168
> URL: https://issues.apache.org/jira/browse/HADOOP-19168
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Rohit Kumar
>Priority: Major
> Labels: pull-request-available
>
> Upgrade Kafka Clients due to CVEs
> CVE-2023-25194:- Affected versions of this package are vulnerable to
> Deserialization of Untrusted Data when there are gadgets in the
> {{{}classpath{}}}. The server will connect to the attacker's LDAP server and
> deserialize the LDAP response, which the attacker can use to execute java
> deserialization gadget chains on the Kafka connect server.
> CVSS Score:- 8.8(High)
> [https://nvd.nist.gov/vuln/detail/CVE-2023-25194]
> CVE-2021-38153
> CVE-2018-17196
> Insufficient Entropy
> [https://security.snyk.io/package/maven/org.apache.kafka:kafka-clients]
> Upgrade Kafka-Clients to 3.4.0 or higher.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19182) Upgrade kafka to 3.4.0
[ https://issues.apache.org/jira/browse/HADOOP-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19182. - Resolution: Duplicate > Upgrade kafka to 3.4.0 > -- > > Key: HADOOP-19182 > URL: https://issues.apache.org/jira/browse/HADOOP-19182 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: fuchaohong >Priority: Major > Labels: pull-request-available > > Upgrade kafka to 3.4.0 to resolve CVE-2023-25194 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19185) Improve ABFS metric integration with iOStatistics
Steve Loughran created HADOOP-19185: --- Summary: Improve ABFS metric integration with iOStatistics Key: HADOOP-19185 URL: https://issues.apache.org/jira/browse/HADOOP-19185 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Steve Loughran Followup to HADOOP-18325 covering the outstanding comments of https://github.com/apache/hadoop/pull/6314/files -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18325) ABFS: Add correlated metric support for ABFS operations
[ https://issues.apache.org/jira/browse/HADOOP-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18325. - Fix Version/s: 3.5.0 Resolution: Fixed > ABFS: Add correlated metric support for ABFS operations > --- > > Key: HADOOP-18325 > URL: https://issues.apache.org/jira/browse/HADOOP-18325 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.3 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add metrics related to a particular job, specific to number of total > requests, retried requests, retry count and others -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19163) Upgrade protobuf version to 3.25.3
[ https://issues.apache.org/jira/browse/HADOOP-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19163. - Resolution: Fixed done. not sure what version to tag with. Proposed: we cut a new release of this > Upgrade protobuf version to 3.25.3 > -- > > Key: HADOOP-19163 > URL: https://issues.apache.org/jira/browse/HADOOP-19163 > Project: Hadoop Common > Issue Type: Bug > Components: hadoop-thirdparty >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19181) IAMCredentialsProvider throttle failures
Steve Loughran created HADOOP-19181:
---
Summary: IAMCredentialsProvider throttle failures
Key: HADOOP-19181
URL: https://issues.apache.org/jira/browse/HADOOP-19181
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Tests report throttling errors in IAM being remapped to noauth and failure
Again, impala tests, but with multiple processes on same host. this means that
HADOOP-18945 isn't sufficient as even if it ensures a singleton instance for a
process
* it doesn't if there are many test buckets (fixable)
* it doesn't work across processes (not fixable)
we may be able to
* use a singleton across all filesystem instances
* once we know how throttling is reported, handle it through retries +
error/stats collection
{code}
2024-02-17T18:02:10,175 WARN [TThreadPoolServer WorkerProcess-22]
fs.FileSystem: Failed to initialize fileystem
s3a://impala-test-uswest2-1/test-warehouse/test_num_values_def_levels_mismatch_15b31ddb.db/too_many_def_levels:
java.nio.file.AccessDeniedException: impala-test-uswest2-1:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
2024-02-17T18:02:10,175 ERROR [TThreadPoolServer WorkerProcess-22]
utils.MetaStoreUtils: Got exception: java.nio.file.AccessDeniedException
impala-test-uswest2-1: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No
AWS Credentials provided by TemporaryAWSCredentialsProvider
SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
java.nio.file.AccessDeniedException: impala-test-uswest2-1:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.maybeTranslateCredentialException(AWSCredentialProviderList.java:351)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$2(S3AFileSystem.java:972)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2748)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:970)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:859)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:715)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.FileSystem.access
[jira] [Resolved] (HADOOP-19172) Upgrade aws-java-sdk to 1.12.720
[ https://issues.apache.org/jira/browse/HADOOP-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19172. - Fix Version/s: 3.3.9 3.5.0 3.4.1 Resolution: Fixed > Upgrade aws-java-sdk to 1.12.720 > > > Key: HADOOP-19172 > URL: https://issues.apache.org/jira/browse/HADOOP-19172 > Project: Hadoop Common > Issue Type: Improvement > Components: build, fs/s3 >Affects Versions: 3.4.0, 3.3.6 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > Update to the latest AWS SDK, to stop anyone worrying about the ion library > CVE https://nvd.nist.gov/vuln/detail/CVE-2024-21634 > This isn't exposed in the s3a client, but may be used downstream. > on v2 sdk releases, the v1 sdk is only used during builds; 3.3.x it is shipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19073) WASB: Fix connection leak in FolderRenamePending
[ https://issues.apache.org/jira/browse/HADOOP-19073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19073. - Resolution: Fixed > WASB: Fix connection leak in FolderRenamePending > > > Key: HADOOP-19073 > URL: https://issues.apache.org/jira/browse/HADOOP-19073 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.6 >Reporter: xy >Assignee: xy >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Fix connection leak in FolderRenamePending in getting bytes -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19176) S3A Xattr headers need hdfs-compatible prefix
Steve Loughran created HADOOP-19176:
---
Summary: S3A Xattr headers need hdfs-compatible prefix
Key: HADOOP-19176
URL: https://issues.apache.org/jira/browse/HADOOP-19176
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
x3a xattr list needs a prefix compatible with hdfs or existing code which tries
to copy attributes between stores can break
we need a prefix of {user/trusted/security/system/raw}.
now, problem: currently xattrs are used by the magic committer to propagate
file size progress; renaming the prefix will break existing code. But as it's
read only we could modify spark to look for both old and new values.
{code}
org.apache.hadoop.HadoopIllegalArgumentException: An XAttr name must be
prefixed with user/trusted/security/system/raw, followed by a '.'
at org.apache.hadoop.hdfs.XAttrHelper.buildXAttr(XAttrHelper.java:77)
at org.apache.hadoop.hdfs.DFSClient.setXAttr(DFSClient.java:2835)
at
org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setXAttr(DistributedFileSystem.java:3115)
at org.apache.hadoop.fs.FileSystem.setXAttr(FileSystem.java:3097)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18958) Improve UserGroupInformation debug log
[ https://issues.apache.org/jira/browse/HADOOP-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18958. - Fix Version/s: 3.5.0 Resolution: Fixed > Improve UserGroupInformation debug log > --- > > Key: HADOOP-18958 > URL: https://issues.apache.org/jira/browse/HADOOP-18958 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.0, 3.3.5 >Reporter: wangzhihui >Assignee: wangzhihui >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > Attachments: 20231029-122825-1.jpeg, 20231029-122825.jpeg, > 20231030-143525.jpeg, image-2023-10-29-09-47-56-489.png, > image-2023-10-30-14-35-11-161.png > > Original Estimate: 1h > Remaining Estimate: 1h > > Using “new Exception( )” to print the call stack of "doAs Method " in > the UserGroupInformation class. Using this way will print meaningless > Exception information and too many call stacks, This is not conducive to > troubleshooting > *example:* > !20231029-122825.jpeg|width=991,height=548! > > *improved result* : > > !image-2023-10-29-09-47-56-489.png|width=1099,height=156! > !20231030-143525.jpeg|width=572,height=674! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18958) UserGroupInformation debug log improve
[ https://issues.apache.org/jira/browse/HADOOP-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-18958: - > UserGroupInformation debug log improve > -- > > Key: HADOOP-18958 > URL: https://issues.apache.org/jira/browse/HADOOP-18958 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.0, 3.3.5 >Reporter: wangzhihui >Priority: Minor > Labels: pull-request-available > Attachments: 20231029-122825-1.jpeg, 20231029-122825.jpeg, > 20231030-143525.jpeg, image-2023-10-29-09-47-56-489.png, > image-2023-10-30-14-35-11-161.png > > Original Estimate: 1h > Remaining Estimate: 1h > > Using “new Exception( )” to print the call stack of "doAs Method " in > the UserGroupInformation class. Using this way will print meaningless > Exception information and too many call stacks, This is not conducive to > troubleshooting > *example:* > !20231029-122825.jpeg|width=991,height=548! > > *improved result* : > > !image-2023-10-29-09-47-56-489.png|width=1099,height=156! > !20231030-143525.jpeg|width=572,height=674! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19175) update s3a committer docs
Steve Loughran created HADOOP-19175: --- Summary: update s3a committer docs Key: HADOOP-19175 URL: https://issues.apache.org/jira/browse/HADOOP-19175 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Update s3a committer docs * declare that magic committer is stable and make it the recommended one * show how to use new command "mapred successfile" to print the success file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: dependabot bumps
thanks. it's more for tracking than anything else. On Mon, 13 May 2024 at 15:52, slfan1989 wrote: > Sorry, I approved these 2 prs. > > In the subsequent process, I will adhere to the common standards. I > overlooked the Jira ID and LICENSE-binary for "Bump org.apache.derby:derby > in /hadoop-project (#6816)". > > I will reapply for a Jira ticket and submit a fix pr. > > > > > hey, who has been approving dependabot bump prs > > ``` > > * b5a90d9500f1 G - Bump org.apache.derby:derby in /hadoop-project (#6816) > > (8 hours ago) > > * 1d09a64e3422 G - Bump org.bouncycastle:bcprov-jdk18on in > /hadoop-project > > (#6811) (23 hours ago) > > ``` > > > these should all be done as real hadoop PRs with JIRA ID and yetus runs, > > including updating the LICENSE-binary files. In particular, HADOOP-19154 > > already was doing the bouncy update to 1.78.1. > > > I'm going to merge that #6755 PR in and for backport do a squashed commit > > of #6811 #6755. >
[jira] [Created] (HADOOP-19172) Upgrade aws-java-sdk to 1.12.720
Steve Loughran created HADOOP-19172: --- Summary: Upgrade aws-java-sdk to 1.12.720 Key: HADOOP-19172 URL: https://issues.apache.org/jira/browse/HADOOP-19172 Project: Hadoop Common Issue Type: Improvement Components: build, fs/s3 Affects Versions: 3.3.6, 3.4.0 Reporter: Steve Loughran Update to the latest AWS SDK, to stop anyone worrying about the ion library CVE https://nvd.nist.gov/vuln/detail/CVE-2024-21634 This isn't exposed in the s3a client, but may be used downstream. on v2 sdk releases, the v1 sdk is only used during builds; 3.3.x it is shipped -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19171) AWS v2: handle alternative forms of connection failure
Steve Loughran created HADOOP-19171:
---
Summary: AWS v2: handle alternative forms of connection failure
Key: HADOOP-19171
URL: https://issues.apache.org/jira/browse/HADOOP-19171
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
We've had reports of network connection failures surfacing deeper in the stack
where we don't convert to AWSApiCallTimeoutException so they aren't retried
properly (retire connection and repeat)
{code}
Unable to execute HTTP request: Broken pipe (Write failed)
{code}
{code}
Your socket connection to the server was not read from or written to within
the timeout period. Idle connections will be closed. (Service: Amazon S3;
Status Code: 400; Error Code: RequestTimeout
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
dependabot bumps
hey, who has been approving dependabot bump prs ``` * b5a90d9500f1 G - Bump org.apache.derby:derby in /hadoop-project (#6816) (8 hours ago) * 1d09a64e3422 G - Bump org.bouncycastle:bcprov-jdk18on in /hadoop-project (#6811) (23 hours ago) ``` these should all be done as real hadoop PRs with JIRA ID and yetus runs, including updating the LICENSE-binary files. In particular, HADOOP-19154 already was doing the bouncy update to 1.78.1. I'm going to merge that #6755 PR in and for backport do a squashed commit of #6811 #6755.
[jira] [Created] (HADOOP-19161) S3A: support a comma separated list of performance flags
Steve Loughran created HADOOP-19161: --- Summary: S3A: support a comma separated list of performance flags Key: HADOOP-19161 URL: https://issues.apache.org/jira/browse/HADOOP-19161 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.4.1 Reporter: Steve Loughran Assignee: Steve Loughran HADOOP-19072 shows we want to add more optimisations than that of HADOOP-18930. * Extending the new optimisations to the existing option is brittle * Adding explicit options for each feature gets complext fast. Proposed * A new class S3APerformanceFlags keeps all the flags * it build this from a string[] of values, which can be extracted from getConf(), * and it can also support a "*" option to mean "everything" * this class can also be handed off to hasPathCapability() and do the right thing. Proposed optimisations * create file (we will hook up HADOOP-18930) * mkdir (HADOOP-19072) * delete (probe for parent path) * rename (probe for source path) We could think of more, with different names, later. The goal is make it possible to strip out every HTTP request we do for safety/posix compliance, so applications have the option of turning off what they don't need. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
whatever happened to python3 from our yetus runs?
builds seem to be failing as python isn't there any more., e.g https://github.com/apache/hadoop/pull/6755: [INFO] --- exec-maven-plugin:1.3.1:exec (shelldocs) @ hadoop-common --- tar: apache-yetus-0.14.0/lib/precommit/qbt.sh: Cannot create symlink to 'test-patch.sh': No such file or directory tar: Exiting with failure status due to previous errors /usr/bin/env: 'python3': No such file or directory
[jira] [Resolved] (HADOOP-19146) noaa-cors-pds bucket access with global endpoint fails
[
https://issues.apache.org/jira/browse/HADOOP-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19146.
-
Fix Version/s: 3.5.0
3.4.1
Resolution: Fixed
> noaa-cors-pds bucket access with global endpoint fails
> --
>
> Key: HADOOP-19146
> URL: https://issues.apache.org/jira/browse/HADOOP-19146
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> All tests accessing noaa-cors-pds use us-east-1 region, as configured at
> bucket level. If global endpoint is configured (e.g. us-west-2), they fail to
> access to bucket.
>
> Sample error:
> {code:java}
> org.apache.hadoop.fs.s3a.AWSRedirectException: Received permanent redirect
> response to region [us-east-1]. This likely indicates that the S3 region
> configured in fs.s3a.endpoint.region does not match the AWS region containing
> the bucket.: null (Service: S3, Status Code: 301, Request ID:
> PMRWMQC9S91CNEJR, Extended Request ID:
> 6Xrg9thLiZXffBM9rbSCRgBqwTxdLAzm6OzWk9qYJz1kGex3TVfdiMtqJ+G4vaYCyjkqL8cteKI/NuPBQu5A0Q==)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:253)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4041)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3947)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$26(S3AFileSystem.java:3924)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3922)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$globStatus$35(S3AFileSystem.java:4956)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2716)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2735)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:4949)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:313)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:281)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:445)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:311)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:328)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:201)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1677)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1674)
> {code}
> {code:java}
> Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null
> (Service: S3, Status Code: 301, Request ID: PMRWMQC9S91CNEJR, Extended
> Request ID:
> 6Xrg9thLiZXffBM9rbSCRgBqwTxdLAzm6OzWk9qYJz1kGex3TVfdiMtqJ+G4vaYCyjkqL8cteKI/NuPBQu5A0Q==)
> at
> software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
> at
> software.amazon.awssdk.pro
[jira] [Resolved] (HADOOP-19159) Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads
[ https://issues.apache.org/jira/browse/HADOOP-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19159. - Fix Version/s: 3.3.9 3.5.0 3.4.1 Resolution: Fixed > Fix hadoop-aws document for fs.s3a.committer.abort.pending.uploads > -- > > Key: HADOOP-19159 > URL: https://issues.apache.org/jira/browse/HADOOP-19159 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Xi Chen >Assignee: Xi Chen >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > The description about `fs.s3a.committer.abort.pending.uploads` in the > _Concurrent Jobs writing to the same destination_ is not all correct. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: magic s3a commiter resilience to failure during task commit
i can see the value in their change "ensures that a job fails when the
committer doesn't do atomic task commit",
But as i believe committers should do exactly that, always, it should
generally be the default.
really they should be able to ask the committer if it has atomic task
commit.
if spark used the existing StreamCapabilities.hasCapability() then we could
have the hadoop bundled committers return true for some probe "atomic task
commit", which the s3a, manifest and v1 file output committer would do
(though there it should check to see if the fs url isn't gcs, s3, s3a,
wasb, ...)
On Wed, 24 Apr 2024 at 00:49, Dylan McClelland
wrote:
> thanks for clarifying that, now that you have pointed me in the right
> direction I can see in the code how that works
>
> I will check with the spark team to see if they are open to either
> modifying outputCommitCoordinator to avoid the authorized commiter check
> when using the magic committer, or promote the undocumented
> "spark.hadoop.outputCommitCoordination.enabled" setting to a fully
> supported option
>
> thanks again
>
> On Tue, Apr 23, 2024 at 8:49 AM Steve Loughran >
> wrote:
>
> > On Sat, 20 Apr 2024 at 23:36, Dylan McClelland
> > wrote:
> >
> > > when describing the resilience of the magic committer to failures
> during
> > a
> > > task commit, the docs state
> > >
> > > "If the .pendingset file has been saved to the job attempt directory,
> the
> > > task has effectively committed, it has just failed to report to the
> > > controller. This will cause complications during job commit, as there
> may
> > > be two task PendingSet committing the same files, or committing files
> > with
> > >
> >
> >
> > last big review/writeup of the protocol was in
> >
> >
> https://github.com/steveloughran/zero-rename-committer/releases/tag/tag_release_2021-05-17
> >
> > when we commit a task the .pendingset file is saved with task id, not
> task
> > attempt id.
> > two task attempts for the same task cannot be committed; the latest one
> to
> > commit wins.
> >
> > That is: task ID is implicit in the filename.
> >
> > There's still the risk of partitioning where TA1 is told to commit, but
> > goes offline, TA2 commits and then, later TA1 completes.
> > Then when the job commits, it gets the files of TA1 even though TA2 is
> the
> > one which reported success.
> >
> >
> > This "fail if task commit doesn't respond/report success" should only be
> > needed for commit protocols whose task commit is
> non-atomic/non-repeatable.
> >
> >
> > > *Proposed*: track task ID in pendingsets, recognise duplicates on load
> > and
> > > then respond by cancelling one set and committing the other. (or
> fail?)"
> > >
> >
> > task id is implicit in filename: it's ${taskid}.pendingset.
> >
> > we can't guarantee which of two TAs told to commit it is, except that the
> > last one told to execute by spark will overwrite any file it found and
> > reported success.
> >
> >
> > >
> > > As far as I can tell from reading over the code, the proposal was not
> > > implemented. Is this still considered a viable solution? If so, I'd be
> > > happy to take a crack at it
> > >
> > > For context: We are running spark jobs using the magic committer in a
> k8s
> > > environment where executor loss is somewhat common (maybe about 2% of
> > > executors are terminated by the cluster). By default, spark's
> > > OutputCommitCoordinator fails the whole write stage (and parent job) if
> > an
> > > executor fails while it an "authorized committer" (see
> > > https://issues.apache.org/jira/browse/SPARK-39195).
> > >
> > > This results in expensive job failures at the final write stage. We can
> > > avoid those failures by disabling the outputCommitCoordinator entirely,
> > but
> > > that leaves us open to the failure mode described above in the s3a
> docs.
> > >
> >
> > I'm not sure you are .
> >
> > >
> > > It seems to me that if the magic committer implemented the above
> proposal
> > > to track task ids in the pendingset, the OutputCommitCoordinator would
> no
> > > longer be needed at all. I think this ticket (
> > > https://issues.apache.org/jira/browse/SPARK-19790) discusses similar
> > ideas
> > > about how to make spark's commit coordiantor work with the s3a
> committers
> > >
> >
> &
[jira] [Created] (HADOOP-19158) Support delegating ByteBufferPositionedReadable to vector reads
Steve Loughran created HADOOP-19158: --- Summary: Support delegating ByteBufferPositionedReadable to vector reads Key: HADOOP-19158 URL: https://issues.apache.org/jira/browse/HADOOP-19158 Project: Hadoop Common Issue Type: Sub-task Components: fs, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Make it easy for any stream with vector io to suppor Specifically, ByteBufferPositionedReadable.readFully() is exactly a single range read so is easy to read. the simpler read() call which can return less isn't part of the vector API. Proposed: invoke the readFully() but convert an EOFException to -1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19157) [ABFS] Filesystem contract tests to use methodPath for robust parallel test runs
Steve Loughran created HADOOP-19157: --- Summary: [ABFS] Filesystem contract tests to use methodPath for robust parallel test runs Key: HADOOP-19157 URL: https://issues.apache.org/jira/browse/HADOOP-19157 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure, test Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran hadoop-azure supports parallel test runs, but unlike hadoop-aws, the azure ones are parallelised across methods in the same test suites. this can fail badly where contract tests have hard coded filenames and assume that they can use this across all test cases. Shows up when you are testing on a store with reduced IO capacity triggering retries and making some test cases slower Fix: hadoop-common contract tests to use methodPath() names -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: magic s3a commiter resilience to failure during task commit
On Sat, 20 Apr 2024 at 23:36, Dylan McClelland
wrote:
> when describing the resilience of the magic committer to failures during a
> task commit, the docs state
>
> "If the .pendingset file has been saved to the job attempt directory, the
> task has effectively committed, it has just failed to report to the
> controller. This will cause complications during job commit, as there may
> be two task PendingSet committing the same files, or committing files with
>
last big review/writeup of the protocol was in
https://github.com/steveloughran/zero-rename-committer/releases/tag/tag_release_2021-05-17
when we commit a task the .pendingset file is saved with task id, not task
attempt id.
two task attempts for the same task cannot be committed; the latest one to
commit wins.
That is: task ID is implicit in the filename.
There's still the risk of partitioning where TA1 is told to commit, but
goes offline, TA2 commits and then, later TA1 completes.
Then when the job commits, it gets the files of TA1 even though TA2 is the
one which reported success.
This "fail if task commit doesn't respond/report success" should only be
needed for commit protocols whose task commit is non-atomic/non-repeatable.
> *Proposed*: track task ID in pendingsets, recognise duplicates on load and
> then respond by cancelling one set and committing the other. (or fail?)"
>
task id is implicit in filename: it's ${taskid}.pendingset.
we can't guarantee which of two TAs told to commit it is, except that the
last one told to execute by spark will overwrite any file it found and
reported success.
>
> As far as I can tell from reading over the code, the proposal was not
> implemented. Is this still considered a viable solution? If so, I'd be
> happy to take a crack at it
>
> For context: We are running spark jobs using the magic committer in a k8s
> environment where executor loss is somewhat common (maybe about 2% of
> executors are terminated by the cluster). By default, spark's
> OutputCommitCoordinator fails the whole write stage (and parent job) if an
> executor fails while it an "authorized committer" (see
> https://issues.apache.org/jira/browse/SPARK-39195).
>
> This results in expensive job failures at the final write stage. We can
> avoid those failures by disabling the outputCommitCoordinator entirely, but
> that leaves us open to the failure mode described above in the s3a docs.
>
I'm not sure you are .
>
> It seems to me that if the magic committer implemented the above proposal
> to track task ids in the pendingset, the OutputCommitCoordinator would no
> longer be needed at all. I think this ticket (
> https://issues.apache.org/jira/browse/SPARK-19790) discusses similar ideas
> about how to make spark's commit coordiantor work with the s3a committers
>
Task attempt failure during commit is a troublespot. Task commits should be
repeatable, so that if task attempt TA1 fails *or fails to report an
outcome* then another attempt TA2 must succeed.
FileOutputCommitter v1, S3A and abfs/gcs manifest committers all meet this
guarantee; those which aren't are
* everywhere: v2 file output committer
* gcs: v1 commit algorithm (which requires atomic dir rename)
* I believe: AWS EMR optimised committer.
I think spark is being over-aggressive and that you can disable the
co-ordinator.
Syed Shameerur Rahman has been doing work on the committer with a goal of
caching state in memory rather than s3 and passing it directly to the
driver rather than via files. This would completely eliminate the issue, as
only the successful task attempts can return the data.
[jira] [Resolved] (HADOOP-19102) [ABFS]: FooterReadBufferSize should not be greater than readBufferSize
[ https://issues.apache.org/jira/browse/HADOOP-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19102. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > [ABFS]: FooterReadBufferSize should not be greater than readBufferSize > -- > > Key: HADOOP-19102 > URL: https://issues.apache.org/jira/browse/HADOOP-19102 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > The method `optimisedRead` creates a buffer array of size `readBufferSize`. > If footerReadBufferSize is greater than readBufferSize, abfs will attempt to > read more data than the buffer array can hold, which causes an exception. > Change: To avoid this, we will keep footerBufferSize = > min(readBufferSizeConfig, footerBufferSizeConfig) > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19153) hadoop-common still exports logback as a transitive dependency
Steve Loughran created HADOOP-19153: --- Summary: hadoop-common still exports logback as a transitive dependency Key: HADOOP-19153 URL: https://issues.apache.org/jira/browse/HADOOP-19153 Project: Hadoop Common Issue Type: Bug Components: build, common Affects Versions: 3.4.0 Reporter: Steve Loughran Even though HADOOP-19084 set out to stop it, somehow ZK's declaration of a logback dependency is still contaminating the hadoop-common dependency graph, so causing problems downstream. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19079) HttpExceptionUtils to check that loaded class is really an exception before instantiation
[ https://issues.apache.org/jira/browse/HADOOP-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19079. - Fix Version/s: 3.3.9 3.5.0 3.4.1 Resolution: Fixed > HttpExceptionUtils to check that loaded class is really an exception before > instantiation > - > > Key: HADOOP-19079 > URL: https://issues.apache.org/jira/browse/HADOOP-19079 > Project: Hadoop Common > Issue Type: Task > Components: common, security >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > It can be dangerous taking class names as inputs from HTTP messages even if > we control the source. Issue is in HttpExceptionUtils in hadoop-common > (validateResponse method). > I can provide a PR that will highlight the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19096) [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic
[ https://issues.apache.org/jira/browse/HADOOP-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19096. - Fix Version/s: 3.5.0 Resolution: Fixed > [ABFS] Enhancing Client-Side Throttling Metrics Updation Logic > -- > > Key: HADOOP-19096 > URL: https://issues.apache.org/jira/browse/HADOOP-19096 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.1 >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > ABFS has a client-side throttling mechanism which works on the metrics > collected from past requests made. I requests are getting failed due to > throttling at server, we update our metrics and client side backoff is > calculated based on those metrics. > This PR enhances the logic to decide which requests should be considered to > compute client side backoff interval as follows: > For each request made by ABFS driver, we will determine if they should > contribute to Client-Side Throttling based on the status code and result: > # Status code in 2xx range: Successful Operations should contribute. > # Status code in 3xx range: Redirection Operations should not contribute. > # Status code in 4xx range: User Errors should not contribute. > # Status code is 503: Throttling Error should contribute only if they are > due to client limits breach as follows: > ## 503, Ingress Over Account Limit: Should Contribute > ## 503, Egress Over Account Limit: Should Contribute > ## 503, TPS Over Account Limit: Should Contribute > ## 503, Other Server Throttling: Should not Contribute. > # Status code in 5xx range other than 503: Should not Contribute. > # IOException and UnknownHostExceptions: Should not Contribute. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19098) Vector IO: consistent specified rejection of overlapping ranges
[ https://issues.apache.org/jira/browse/HADOOP-19098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19098. - Resolution: Fixed > Vector IO: consistent specified rejection of overlapping ranges > --- > > Key: HADOOP-19098 > URL: https://issues.apache.org/jira/browse/HADOOP-19098 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/s3 >Affects Versions: 3.3.6 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > Related to PARQUET-2171 q: "how do you deal with overlapping ranges?" > I believe s3a rejects this, but the other impls may not. > Proposed > FS spec to say > * "overlap triggers IllegalArgumentException". > * special case: 0 byte ranges may be short circuited to return empty buffer > even without checking file length etc. > Contract tests to validate this > (+ common helper code to do this). > I'll copy the validation stuff into the parquet PR for consistency with older > releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19101) Vectored Read into off-heap buffer broken in fallback implementation
[
https://issues.apache.org/jira/browse/HADOOP-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19101.
-
Fix Version/s: 3.3.9
3.4.1
Resolution: Fixed
> Vectored Read into off-heap buffer broken in fallback implementation
>
>
> Key: HADOOP-19101
> URL: https://issues.apache.org/jira/browse/HADOOP-19101
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure
>Affects Versions: 3.4.0, 3.3.6
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.3.9, 3.5.0, 3.4.1
>
>
> {{VectoredReadUtils.readInDirectBuffer()}} always starts off reading at
> position zero even when the range is at a different offset. As a result: you
> can get incorrect information.
> Thanks for this is straightforward: we pass in a FileRange and use its offset
> as the starting position.
> However, this does mean that all shipping releases 3.3.5-3.4.0 cannot safely
> read vectorIO into direct buffers through HDFS, ABFS or GCS. Note that we
> have never seen this in production because the parquet and ORC libraries both
> read into on-heap storage.
> Those libraries needs to be audited to make sure that they never attempt to
> read into off-heap DirectBuffers. This is a bit trickier than you would think
> because an allocator is passed in. For PARQUET-2171 we will
> * only invoke the API on streams which explicitly declare their support for
> the API (so fallback in parquet itself)
> * not invoke when direct buffer allocation is in use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19144) S3A prefetching to support Vector IO
Steve Loughran created HADOOP-19144: --- Summary: S3A prefetching to support Vector IO Key: HADOOP-19144 URL: https://issues.apache.org/jira/browse/HADOOP-19144 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Add explicit support for vector IO in s3a prefetching stream. * if a range is in 1+ cached block, it SHALL be read from cache and returned * if a range is not in cache : TBD * If a range is partially in cache: TBD these are the same decisions that abfs has to make: should the client fetch/cache block or just do one or more GET requests A big issue is: does caching of data fetched in a range request make any sense at all? Or more specifically: does fetching the blocks in which range requests are found make sense Simply going to the store is a lot simpler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19140) [ABFS, S3A] Add IORateLimiter api to hadoop common
Steve Loughran created HADOOP-19140: --- Summary: [ABFS, S3A] Add IORateLimiter api to hadoop common Key: HADOOP-19140 URL: https://issues.apache.org/jira/browse/HADOOP-19140 Project: Hadoop Common Issue Type: Sub-task Components: fs, fs/azure, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Create a rate limiter API in hadoop common which code (initially, manifest committer, bulk delete).. can request iO capacity for a specific operation. this can be exported by filesystems so support shared rate limiting across all threads pulled from HADOOP-19093 PR -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19115) upgrade to nimbus-jose-jwt 9.37.2 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19115. - Fix Version/s: 3.3.9 3.5.0 3.4.1 Assignee: PJ Fanning Resolution: Fixed > upgrade to nimbus-jose-jwt 9.37.2 due to CVE > > > Key: HADOOP-19115 > URL: https://issues.apache.org/jira/browse/HADOOP-19115 > Project: Hadoop Common > Issue Type: Bug > Components: build, CVE >Affects Versions: 3.4.0, 3.5.0 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > https://github.com/advisories/GHSA-gvpg-vgmx-xg6w -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19131) Assist reflection iO with WrappedOperations class
Steve Loughran created HADOOP-19131: --- Summary: Assist reflection iO with WrappedOperations class Key: HADOOP-19131 URL: https://issues.apache.org/jira/browse/HADOOP-19131 Project: Hadoop Common Issue Type: Sub-task Components: fs, fs/azure, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran parquet, avro etc are still stuck building with older hadoop releases. This makes using new APIs hard (PARQUET-2117) and means that APIs which are 5 years old (!) such as HADOOP-15229 just aren't picked up. This lack of openFIle() adoption hurts working with files in cloud storage as * extra HEAD requests are made * read policies can't be explicitly set * split start/end can't be passed down Proposed # create class org.apache.hadoop.io.WrappedOperations # add methods to wrap the apis # test in contract tests via reflection loading -verifies we have done it properly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19047) Support InMemory Tracking Of S3A Magic Commits
[ https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19047. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > Support InMemory Tracking Of S3A Magic Commits > -- > > Key: HADOOP-19047 > URL: https://issues.apache.org/jira/browse/HADOOP-19047 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > The following are the operations which happens within a Task when it uses S3A > Magic Committer. > *During closing of stream* > 1. A 0-byte file with a same name of the original file is uploaded to S3 > using PUT operation. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152] > for more information. This is done so that the downstream application like > Spark could get the size of the file which is being written. > 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176] > for more information. > *During TaskCommit* > 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number > of metadata file in S3 if a single task writes to 'x' files) are read and > rewritten to S3 as a single metadata file. Refer > [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201] > for more information > Since these operations happens with the Task JVM, We could optimize as well > as save cost by storing these information in memory when Task memory usage is > not a constraint. Hence the proposal here is to introduce a new MagicCommit > Tracker called "InMemoryMagicCommitTracker" which will store the > 1. Metadata of MPU in memory till the Task is committed > 2. Store the size of the file which can be used by the downstream application > to get the file size before it is committed/visible to the output path. > This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call > given a Task writes only 1 file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19116) update to zookeeper client 3.8.4 due to CVE-2024-23944
[ https://issues.apache.org/jira/browse/HADOOP-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19116. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > update to zookeeper client 3.8.4 due to CVE-2024-23944 > --- > > Key: HADOOP-19116 > URL: https://issues.apache.org/jira/browse/HADOOP-19116 > Project: Hadoop Common > Issue Type: Bug > Components: CVE >Affects Versions: 3.4.0, 3.3.6 >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > https://github.com/advisories/GHSA-r978-9m6m-6gm6 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19089) [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path
[ https://issues.apache.org/jira/browse/HADOOP-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19089. - Fix Version/s: 3.5.0 Resolution: Fixed > [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path > --- > > Key: HADOOP-19089 > URL: https://issues.apache.org/jira/browse/HADOOP-19089 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0, 3.4.1 >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > A while back changes were made to support HDFS.setXAttr() and HDFS.getXAttr() > on root path for ABFS Driver. > For these, filesystem level APIs were introduced and used to set/get metadata > of container. > Refer to Jira: [HADOOP-18869] ABFS: Fixing Behavior of a File System APIs on > root path - ASF JIRA (apache.org) > Ideally, same set of APIs should be used, and root should be treated as a > path like any other path. > This change is to avoid calling container APIs for these HDFS calls. > As a result of this these APIs will fail on root path (as earlier) because > service does not support get/set of user properties on root path. > This change will also update the documentation to reflect that these > operations are not supported on root path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19122) testListPathWithValueGreaterThanServerMaximum assert failure on heavily loaded store
Steve Loughran created HADOOP-19122: --- Summary: testListPathWithValueGreaterThanServerMaximum assert failure on heavily loaded store Key: HADOOP-19122 URL: https://issues.apache.org/jira/browse/HADOOP-19122 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.4.0 Reporter: Steve Loughran on an azure store which may be experiencing throttling. the listPath call returns less than the 5K limit. the assertion needs to be changed for this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19050) Add S3 Access Grants Support in S3A
[ https://issues.apache.org/jira/browse/HADOOP-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19050. - Fix Version/s: 3.5.0 Resolution: Fixed Fixed in trunk; backport to 3.4 should go in later. > Add S3 Access Grants Support in S3A > --- > > Key: HADOOP-19050 > URL: https://issues.apache.org/jira/browse/HADOOP-19050 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Jason Han >Assignee: Jason Han >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > > Add support for S3 Access Grants > (https://aws.amazon.com/s3/features/access-grants/) in S3A. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19119) spotbugs complaining about possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize()
[
https://issues.apache.org/jira/browse/HADOOP-19119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19119.
-
Fix Version/s: 3.5.0
3.4.1
Resolution: Fixed
> spotbugs complaining about possible NPE in
> org.apache.hadoop.crypto.key.kms.ValueQueue.getSize()
>
>
> Key: HADOOP-19119
> URL: https://issues.apache.org/jira/browse/HADOOP-19119
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: crypto
>Affects Versions: 3.5.0
> Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> PRs against hadoop-common are reporting spotbugs problems
> {code}
> Dodgy code Warnings
> Code Warning
> NPPossible null pointer dereference in
> org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return
> value of called method
> Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE (click for details)
> In class org.apache.hadoop.crypto.key.kms.ValueQueue
> In method org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String)
> Local variable stored in JVM register ?
> Dereferenced at ValueQueue.java:[line 332]
> Known null at ValueQueue.java:[line 332]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19119) spotbugs complaining about possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.alueQueue.getSize()
Steve Loughran created HADOOP-19119:
---
Summary: spotbugs complaining about possible NPE in
org.apache.hadoop.crypto.key.kms.ValueQueue.alueQueue.getSize()
Key: HADOOP-19119
URL: https://issues.apache.org/jira/browse/HADOOP-19119
Project: Hadoop Common
Issue Type: Sub-task
Components: crypto
Affects Versions: 3.5.0
Reporter: Steve Loughran
Assignee: Steve Loughran
PRs against hadoop-common are reporting spotbugs problems
{code}
Dodgy code Warnings
CodeWarning
NP Possible null pointer dereference in
org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String) due to return value
of called method
Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE (click for details)
In class org.apache.hadoop.crypto.key.kms.ValueQueue
In method org.apache.hadoop.crypto.key.kms.ValueQueue.getSize(String)
Local variable stored in JVM register ?
Dereferenced at ValueQueue.java:[line 332]
Known null at ValueQueue.java:[line 332]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19066) AWS SDK V2 - Enabling FIPS should be allowed with central endpoint
[ https://issues.apache.org/jira/browse/HADOOP-19066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19066. - Fix Version/s: 3.4.1 Resolution: Fixed > AWS SDK V2 - Enabling FIPS should be allowed with central endpoint > -- > > Key: HADOOP-19066 > URL: https://issues.apache.org/jira/browse/HADOOP-19066 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.5.0, 3.4.1 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > FIPS support can be enabled by setting "fs.s3a.endpoint.fips". Since the SDK > considers overriding endpoint and enabling fips as mutually exclusive, we > fail fast if fs.s3a.endpoint is set with fips support (details on > HADOOP-18975). > Now, we no longer override SDK endpoint for central endpoint since we enable > cross region access (details on HADOOP-19044) but we would still fail fast if > endpoint is central and fips is enabled. > Changes proposed: > * S3A to fail fast only if FIPS is enabled and non-central endpoint is > configured. > * Tests to ensure S3 bucket is accessible with default region us-east-2 with > cross region access (expected with central endpoint). > * Document FIPS support with central endpoint on connecting.html. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-19066) AWS SDK V2 - Enabling FIPS should be allowed with central endpoint
[ https://issues.apache.org/jira/browse/HADOOP-19066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19066: - > AWS SDK V2 - Enabling FIPS should be allowed with central endpoint > -- > > Key: HADOOP-19066 > URL: https://issues.apache.org/jira/browse/HADOOP-19066 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.5.0, 3.4.1 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > FIPS support can be enabled by setting "fs.s3a.endpoint.fips". Since the SDK > considers overriding endpoint and enabling fips as mutually exclusive, we > fail fast if fs.s3a.endpoint is set with fips support (details on > HADOOP-18975). > Now, we no longer override SDK endpoint for central endpoint since we enable > cross region access (details on HADOOP-19044) but we would still fail fast if > endpoint is central and fips is enabled. > Changes proposed: > * S3A to fail fast only if FIPS is enabled and non-central endpoint is > configured. > * Tests to ensure S3 bucket is accessible with default region us-east-2 with > cross region access (expected with central endpoint). > * Document FIPS support with central endpoint on connecting.html. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19108) S3 Express: document use
Steve Loughran created HADOOP-19108: --- Summary: S3 Express: document use Key: HADOOP-19108 URL: https://issues.apache.org/jira/browse/HADOOP-19108 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran The 3.4.0 release doesn't explicitly cover S3 Express. It's support is automatic * library handles it * hadoop shell commands know that there may be "missing" dirs in treewalks due to in-flight uploads * s3afs automatically switches to deleting pending uploads in delete(dir) call. we just need to provide a summary of features, how to probe etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
followup: overnight work happy too. one interesting pain point is that on a raspberry pi 64 os checknative complains that libcrypto is missing > bin/hadoop checknative 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built without PMDK support. 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher. java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)! at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method) at org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90) at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111) Native library checking: hadoop: true /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0 zlib:true /lib/aarch64-linux-gnu/libz.so.1 zstd : true /lib/aarch64-linux-gnu/libzstd.so.1 bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1 openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)! ISA-L: false libhadoop was built without ISA-L support PMDK:false The native code was built without PMDK support. which happens because its not in /lib/aarch64-linux-gnu but instead in /usr/lib/aarch64-linux-gnu/l ls -l /usr/lib/aarch64-linux-gnu/libcrypto* -rw-r--r-- 1 root root 2739952 Sep 19 13:09 /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 -rw-r--r-- 1 root root 4466856 Oct 27 13:40 /usr/lib/aarch64-linux-gnu/libcrypto.so.3 Anyone got any insights on how I should set up this (debian-based) OS here? I know it's only a small box but with arm64 VMs becoming available in cloud infras, it'd be good to know if they are similar. Note: checknative itself is happy; but checknative -a will fail because of this -though it's an OS setup issue, nothing related to the hadoop binaries. steve On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He wrote: > Hi Shilun, Counter should be with yourself vote, where the current summary > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline. > Thanks again. > > Best Regards, > - He Xiaoqiao > > On Tue, Mar 12, 2024 at 9:00 AM slfan1989 wrote: > > > As of now, we have collected 5 affirmative votes, with 4 votes binding > and > > 1 vote non-binding. > > > > Thank you very much for voting and verifying! > > > > This voting will continue until March 15th, this Friday. > > > > Best Regards, > > Shilun Fan. > > > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran > > > > > wrote: > > > > > +1 binding > > > > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen > it > > > earlier. been testing it this afternoon: > > > > > > pulled the latest version of > > > https://github.com/apache/hadoop-release-support > > > (note, this module is commit-then-review; whoever is working > > on/validating > > > a release can commit as they go along. This is not production code...) > > > > > > * went through the "validating a release" step, validating maven > > artifacts > > > * building the same downstream modules which built for me last time > (avro > > > too complex; hboss not aws v2 in apache yet) > > > > > > spark build is still ongoing, but I'm not going to wait. It is > building, > > > which is key. > > > > > > The core changes I needed in are at the dependency level and I've > > > verified they are good. > > > > > > Oh, and I've also got my raspberry p5 doing the download of the arm > > > stuff for its checknative; not expecting problems. > > > > > > So: i've got some stuff still ongoing, but the core changes to > packaging > > > are in and the rest I'm not worried about -they shouldn't block the > > release > > > as I already validated them on RC2 > > > > > > > > > > > > > > > > > > On Mon, 4 Mar 2024 at 22:08, slfan1989 wrote: > > > > > > > Hi folks, > > > > > > > > Xiaoqiao He and I have put together a release candidate (RC3) for > > Hadoop > > > > 3.4.0. > > > > > > > > What we would like is for anyone who can to verify the tarballs, > >
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)
+1 binding (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it earlier. been testing it this afternoon: pulled the latest version of https://github.com/apache/hadoop-release-support (note, this module is commit-then-review; whoever is working on/validating a release can commit as they go along. This is not production code...) * went through the "validating a release" step, validating maven artifacts * building the same downstream modules which built for me last time (avro too complex; hboss not aws v2 in apache yet) spark build is still ongoing, but I'm not going to wait. It is building, which is key. The core changes I needed in are at the dependency level and I've verified they are good. Oh, and I've also got my raspberry p5 doing the download of the arm stuff for its checknative; not expecting problems. So: i've got some stuff still ongoing, but the core changes to packaging are in and the rest I'm not worried about -they shouldn't block the release as I already validated them on RC2 On Mon, 4 Mar 2024 at 22:08, slfan1989 wrote: > Hi folks, > > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop > 3.4.0. > > What we would like is for anyone who can to verify the tarballs, especially > anyone who can try the arm64 binaries as we want to include them too. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/ > > The git tag is release-3.4.0-RC3, commit bd8b77f398f > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1408 > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md > > This is off branch-3.4.0 and is the first big release since 3.3.6. > > Key changes include > > * S3A: Upgrade AWS SDK to V2 > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks > * YARN Federation improvements > * YARN Capacity Scheduler improvements > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes > * HDFS EC: Code Enhancements and Bug Fixes > * Transitive CVE fixes > > Differences from Hadoop-3.4.0-RC2 > > * From branch-3.4 to branch-3.4.0 backport 2 Prs > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b) > * HADOOP-19084: Pruning hadoop-common transitive dependencies. > (80b4bb68159c) > * Use hadoop-release-support[1] for packaging and verification. > * Add protobuf compatibility issue description > > Note, because the arm64 binaries are built separately on a different > platform and JVM, their jar files may not match those of the x86 > release -and therefore the maven artifacts. I don't think this is > an issue (the ASF actually releases source tarballs, the binaries are > there for help only, though with the maven repo that's a bit blurred). > > The only way to be consistent would actually untar the x86.tar.gz, > overwrite its binaries with the arm stuff, retar, sign and push out > for the vote. Even automating that would be risky. > > [1] hadoop-release-support: > https://github.com/apache/hadoop-release-support > Thanks to steve for providing hadoop-release-support. > > Best Regards, > Shilun Fan. > >
Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop
+1 for cutting hbase 1; it only reduces dependency pain (no more protobuf 2.5!) Created the JIRA on that a few days back https://issues.apache.org/jira/browse/YARN-11658 On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault wrote: > Hbase v1 is EOL for a while now, so option 2 probably makes sense. While > you are at it you should probably update the hbase2 version, because 2.2.x > is also very old and EOL. 2.5.x is the currently maintained release for > hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as > well. > > On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena wrote: > > > Hi Folks, > > As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other > > for v2(2.2.4). The versions are specified over here: [1], how to build is > > mentioned over here: [2] > > > > As of now we by default run our Jenkins "only" for HBase v1, so we have > > seen HBase v2 profile silently breaking a couple of times. > > > > Considering there are stable versions for HBase v2 as per [3] & HBase v2 > > seems not too new, I have some suggestions, we can consider: > > > > * Make HBase v2 profile as the default profile & let HBase v1 profile > stay > > in our code. > > * Ditch HBase v1 profile & just lets support HBase v2 profile. > > * Let everything stay as is, just add a Jenkins job/ Github action which > > compiles HBase v2 as well, so we make sure no change breaks it. > > > > Personally I would go with the second option, the last HBase v1 release > > seems to be 2 years back, it might be pulling in some > > problematic transitive dependencies & it will open scope for us to > support > > HBase 3.x when they have a stable release in future. > > > > > > Let me know your thoughts!!! > > > > -Ayush > > > > > > [1] > > > > > https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207 > > > > [2] > > > > > https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172 > > > > [3] https://hbase.apache.org/downloads.html > > >
[jira] [Created] (HADOOP-19105) S3A: Recover from Vector IO read failures
Steve Loughran created HADOOP-19105: --- Summary: S3A: Recover from Vector IO read failures Key: HADOOP-19105 URL: https://issues.apache.org/jira/browse/HADOOP-19105 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.6, 3.4.0 Environment: s3a vector IO doesn't try to recover from read failures the way read() does. Need to * abort HTTP stream if considered needed * retry active read which failed * but not those which had succeeded Reporter: Steve Loughran -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
[
https://issues.apache.org/jira/browse/HADOOP-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19043.
-
Fix Version/s: 3.5.0
Resolution: Fixed
> S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
> -
>
> Key: HADOOP-19043
> URL: https://issues.apache.org/jira/browse/HADOOP-19043
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Getting test failures in the new ITestS3AOpenCost tests when run with
> {{-Dprefetch}}
> Thought I'd tested this, but clearly not
> * class cast failures on asserts (fix: skip)
> * bytes read different in one test: (fix: identify and address)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19104) S3A HeaderProcessing to process all metadata entries of HEAD response
Steve Loughran created HADOOP-19104:
---
Summary: S3A HeaderProcessing to process all metadata entries of
HEAD response
Key: HADOOP-19104
URL: https://issues.apache.org/jira/browse/HADOOP-19104
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
S3A HeaderProcessing builds up an incomplete list of headers as its mapping of
md to header. entries omits headers including
x-amz-server-side-encryption-aws-kms-key-id
proposed
* review all headers which are stripped from "raw" responses and mapped into
headers
* make sure result of headers matches v1; looks like etags are different
* make sure x-amz-server-side-encryption-aws-kms-key-id gets back
* plus new checksum values
{code}
v1 sdk
{code}
# file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
header.Content-Length="524671"
header.Content-Type="binary/octet-stream"
header.ETag="3e39531220fbd3747d32cf93a79a7a0c"
header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
header.x-amz-server-side-encryption="AES256"
{code}
v2 SDK. note how etag is now double quoted.
{code}
# file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
header.Content-Length="524671"
header.Content-Type="binary/octet-stream"
header.ETag=""3e39531220fbd3747d32cf93a79a7a0c""
header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
header.x-amz-server-side-encryption="AES256"
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19097) core-default fs.s3a.connection.establish.timeout value too low -warning always printed
[
https://issues.apache.org/jira/browse/HADOOP-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19097.
-
Fix Version/s: 3.5.0
3.4.1
Resolution: Fixed
> core-default fs.s3a.connection.establish.timeout value too low -warning
> always printed
> --
>
> Key: HADOOP-19097
> URL: https://issues.apache.org/jira/browse/HADOOP-19097
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
> Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> caused by HADOOP-18915.
> in core-default we set the value of fs.s3a.connection.establish.timeout to 5s
> {code}
>
> fs.s3a.connection.establish.timeout
> 5s
>
> {code}
> but there is a minimum of 15s, so this prints a warning
> {code}
> 2024-02-29 10:39:27,369 WARN impl.ConfigurationHelper: Option
> fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000
> ms instead
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19082) S3A: Update AWS SDK V2 to 2.24.6
[ https://issues.apache.org/jira/browse/HADOOP-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19082. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > S3A: Update AWS SDK V2 to 2.24.6 > > > Key: HADOOP-19082 > URL: https://issues.apache.org/jira/browse/HADOOP-19082 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Harshit Gupta >Assignee: Harshit Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging > w.r.t. imds module. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19101) Vectored Read into off-heap buffer broken
Steve Loughran created HADOOP-19101:
---
Summary: Vectored Read into off-heap buffer broken
Key: HADOOP-19101
URL: https://issues.apache.org/jira/browse/HADOOP-19101
Project: Hadoop Common
Issue Type: Sub-task
Components: fs, fs/azure
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
{{VectoredReadUtils.readInDirectBuffer()}} always starts off reading at
position zero even when the range is at a different offset. As a result: you
can get incorrect information.
Thanks for this is straightforward: we pass in a FileRange and use its offset
as the starting position.
However, this does mean that all shipping releases 3.3.5-3.4.0 cannot safely
read vectorIO into direct buffers through HDFS, ABFS or Azure. Note that we
have never seen this in production because the parquet and ORC libraries both
read into on-heap storage.
Those libraries needs to be audited to make sure that they never attempt to
read into off-heap DirectBuffers. This is a bit trickier than you would think
because an allocator is passed in. For PARQUET-2171 we will
* only invoke the API on streams which explicitly declare their support for the
API (so fallback in parquet itself)
* not invoke when direct buffer allocation is in use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19098) Vector IO: consistent specified rejection of overlapping ranges
Steve Loughran created HADOOP-19098: --- Summary: Vector IO: consistent specified rejection of overlapping ranges Key: HADOOP-19098 URL: https://issues.apache.org/jira/browse/HADOOP-19098 Project: Hadoop Common Issue Type: Improvement Components: fs, fs/s3 Affects Versions: 3.3.6 Reporter: Steve Loughran Assignee: Steve Loughran Related to PARQUET-2171 q: "how do you deal with overlapping ranges?" I believe s3a rejects this, but the other impls may not. Proposed FS spec to say * "overlap triggers IllegalArgumentException". * special case: 0 byte ranges may be short circuited to return empty buffer even without checking file length etc. Contract tests to validate this (+ common helper code to do this). I'll copy the validation stuff into the parquet PR for consistency with older releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19097) core-default fs.s3a.connection.establish.timeout value too low -warning always printed
Steve Loughran created HADOOP-19097:
---
Summary: core-default fs.s3a.connection.establish.timeout value
too low -warning always printed
Key: HADOOP-19097
URL: https://issues.apache.org/jira/browse/HADOOP-19097
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
caused by HADOOP-18915.
in core-default we set the value of fs.s3a.connection.establish.timeout to 5s
{code}
fs.s3a.connection.establish.timeout
5s
{code}
but there is a minimum of 15s, so this prints a warning
{code}
2024-02-29 10:39:27,369 WARN impl.ConfigurationHelper: Option
fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms
instead
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19095) hadoop-aws: downgrade openssl export to test
Steve Loughran created HADOOP-19095: --- Summary: hadoop-aws: downgrade openssl export to test Key: HADOOP-19095 URL: https://issues.apache.org/jira/browse/HADOOP-19095 Project: Hadoop Common Issue Type: Improvement Components: build, fs/s3 Affects Versions: 3.3.4, 3.3.3, 3.3.5, 3.3.2, 3.3.1, 3.3.0, 3.4.0 Reporter: Steve Loughran As seen in dependency scans and mentioned in HADOOP-16346; wildfly/openssl jar is exported as runtime; it is only needed at test. proposed: downgrade -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19093) add load tests for abfs rename resilience
Steve Loughran created HADOOP-19093: --- Summary: add load tests for abfs rename resilience Key: HADOOP-19093 URL: https://issues.apache.org/jira/browse/HADOOP-19093 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure, test Affects Versions: 3.3.6 Reporter: Steve Loughran Assignee: Steve Loughran I need a load test to verify that the rename resilience of the manifest committer actually works as intended * test suite with name ILoadTest* prefix (as with s3) * parallel test running with many threads doing many renames * verify that rename recovery should be detected * and that all renames MUST NOT fail. maybe also: metrics for this in fs and doc update. Possibly; LogExactlyOnce to warn of load issues -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19092) ABFS phase 4: post Hadoop 3.4.0 features
Steve Loughran created HADOOP-19092: --- Summary: ABFS phase 4: post Hadoop 3.4.0 features Key: HADOOP-19092 URL: https://issues.apache.org/jira/browse/HADOOP-19092 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.4.0 Reporter: Steve Loughran Uber-JIRA for ABFS work so we can close HADOOP-18072 as done for 3.4.0 Assuming 3.4.1 is a rapid roll of packing, dependencies and critical fixes, this should target 3.4.2 and beyond -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)
I have been testing this all week, and a -1 until some very minor changes go in. 1. build the arm64 binaries with the same jar artifacts as the x86 one 2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j. 3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive dependencies For #1 we have automation there in my client-validator module, which I have moved to be a hadoop-managed project and tried to make more manageable https://github.com/apache/hadoop-release-support This contains an ant project to perform a lot of the documented build stages, including using SCP to copy down an x86 release tarball and make a signed copy of this containing (locally built) arm artifacts. Although that only works with my development environment (macbook m1 laptop and remote ec2 server), it should be straightforward to make it more flexible. It also includes and tests a maven project which imports many of the hadoop-* pom files and run some test with it; this caught some problems with exported slf4j and log4j2 artifacts getting into the classpath. That is: hadoop-common pulling in log4j 1.2 and 2.x bindings. HADOOP-19084 fixes this; the build file now includes a target to scan the dependencies and fail if "forbidden" artifacts are found. I have not been able to stop logback ending on the transitive dependency list, but at least there is only one slf4j there. HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j while the move to v2 is still something we have to consider a WiP. I have tried doing some other changes to the packaging this week - creating a lean distro without the AWS SDK - trying to get protobuf-2.5 out of yarn-api However, I think it is too late to try applying patches this risky. I Believe we should get the 3.4.0 release out for people to start playing with while we rapidly iterate 3.4.1 release out with - updated dependencies (where possible) - separate "lean" and "full" installations, where "full" includes all the cloud connectors and their dependencies; the default is lean and doesn't. That will cut the default download size in half. - critical issues which people who use the 3.4.0 release raise with us. That is: a packaging and bugs release, with a minimal number of new features. I've created HADOOP-19087 <https://issues.apache.org/jira/browse/HADOOP-19087> to cover this, I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have put a lot of work on 3.4.0 and probably need other people to take up the work for next release. Who else is willing to participate? (Yes Mukund, I have you in mind too) One thing I would like to visit is: what hadoop-tools modules can we cut? Are rumen and hadoop-streaming being actively used? Or can we consider them implicitly EOL and strip. Just think of the maintenance effort we would save. --- Incidentally, I have tested the arm stuff on my raspberry pi5 which is now running 64 bit linux. I believe it is the first time we have qualified a Hadoop release with the media player under someone's television. On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur wrote: > Thanks, Shilun for putting this together. > > Tried the below things and everything worked for me. > > validated checksum and gpg signature. > compiled from source. > Ran AWS integration tests. > untar the binaries and able to access objects in S3 via hadoop fs commands. > compiled gcs-connector successfully using the 3.4.0 version. > > qq: what is the difference between RC1 and RC2? apart from some extra > patches. > > > > On Thu, Feb 15, 2024 at 10:58 AM slfan1989 wrote: > >> Thank you for explaining this part! >> >> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to >> generate >> the ARM tar package, which should meet expectations. >> >> We also look forward to other members helping to verify. >> >> Best Regards, >> Shilun Fan. >> >> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran >> wrote: >> >> > >> > >> > On Mon, 12 Feb 2024 at 15:32, slfan1989 wrote: >> > >> >> >> >> >> >> Note, because the arm64 binaries are built separately on a different >> >> platform and JVM, their jar files may not match those of the x86 >> >> release -and therefore the maven artifacts. I don't think this is >> >> an issue (the ASF actually releases source tarballs, the binaries are >> >> there for help only, though with the maven repo that's a bit blurred). >> >> >> >> The only way to be consistent would actually untar the x86.tar.gz, >> >> overwrite its binaries with the arm stuff, retar, sign and push out >> >> for the vote. >> > >> > >> >
[jira] [Created] (HADOOP-19087) Release Hadoop 3.4.1
Steve Loughran created HADOOP-19087: --- Summary: Release Hadoop 3.4.1 Key: HADOOP-19087 URL: https://issues.apache.org/jira/browse/HADOOP-19087 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0 Reporter: Steve Loughran Release a minor update to hadoop 3.4.0 with * packaging enhancements * updated dependencies (where viable) * fixes for critical issues found after 3.4.0 released * low-risk feature enhancements (those which don't impact schedule...) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19065) Update Protocol Buffers installation to 3.21.12
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19065. - Fix Version/s: 3.5.0 Resolution: Fixed > Update Protocol Buffers installation to 3.21.12 > > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: huangzhaobo >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Update docs and docker script to cover downloading the 3.21.12 protobuf > compiler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19086) move commons-logging to 1.2
Steve Loughran created HADOOP-19086: --- Summary: move commons-logging to 1.2 Key: HADOOP-19086 URL: https://issues.apache.org/jira/browse/HADOOP-19086 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0 Reporter: Steve Loughran although hadoop doesn't use the APIs itself, it bundles commons-logging as things it depends on (http components) do. the version hadoop declares (1.1.3) is out of date compared to its dependencies * update pom and LICENSE-binary -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18487) Make protobuf 2.5 an optional runtime dependency.
[ https://issues.apache.org/jira/browse/HADOOP-18487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-18487: - still there under yarn-api; will do followup > Make protobuf 2.5 an optional runtime dependency. > - > > Key: HADOOP-18487 > URL: https://issues.apache.org/jira/browse/HADOOP-18487 > Project: Hadoop Common > Issue Type: Improvement > Components: build, ipc >Affects Versions: 3.3.4 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > uses of protobuf 2.5 and RpcEnginej have been deprecated since 3.3.0 in > HADOOP-17046 > while still keeping those files around (for a long time...), how about we > make the protobuf 2.5.0 export off hadoop common and hadoop-hdfs *provided*, > rather than *compile* > that way, if apps want it for their own apis, they have to explicitly ask for > it, but at least our own scans don't break. > i have no idea what will happen to the rest of the stack at this point, it > will be "interesting" to see -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19084) hadoop-common exports logback as a transitive dependency
Steve Loughran created HADOOP-19084:
---
Summary: hadoop-common exports logback as a transitive dependency
Key: HADOOP-19084
URL: https://issues.apache.org/jira/browse/HADOOP-19084
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.4.0, 3.5.0
Reporter: Steve Loughran
this is probably caused by HADOOP-18613:
ZK is pulling in some extra transitive stuff which surfaces in applications
which import hadoop-common into their poms. It doesn't seem to show up in our
distro, but downstream you get warnings about duplicate logging stuff
{code}
| +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
| | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
| | | \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile - omitted
for duplicate)
| | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
| | +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for
conflict with 4.1.100.Final)
| | +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile -
omitted for conflict with 4.1.100.Final)
| | +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
| | +- ch.qos.logback:logback-core:jar:1.2.10:compile
| | +- ch.qos.logback:logback-classic:jar:1.2.10:compile
| | | +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for
duplicate)
| | | \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict with
1.7.30)
| | \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict with
2.14.0)
{code}
proposed: exclude the zk dependencies we either override outselves or don't
need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19083) hadoop binary tarball to exclude aws v2 sdk
Steve Loughran created HADOOP-19083: --- Summary: hadoop binary tarball to exclude aws v2 sdk Key: HADOOP-19083 URL: https://issues.apache.org/jira/browse/HADOOP-19083 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Have the default hadoop binary .tar.gz exclude the aws v2 sdk by default. This SDK brings the total size of the distribution to about 1 GB. Proposed * add a profile to include the aws sdk in the dist module * disable it by default Instead we document which version is needed. The hadoop-aws and hadoop-cloud storage maven artifacts will declare their dependencies, so apps building with those get to do the download. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19080) S3A createFakeDirectory/put fails on object lock bucket if the path exists
Steve Loughran created HADOOP-19080: --- Summary: S3A createFakeDirectory/put fails on object lock bucket if the path exists Key: HADOOP-19080 URL: https://issues.apache.org/jira/browse/HADOOP-19080 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.3.6 Reporter: Steve Loughran s3 bucket with object lock enabled fails in createFakeDirectory (reported on S.O) error implies that we need to calculate and include the md5 checksum on the PUT, which gets complex once you include CSE into the mix: the checksum of the encrypted data is what'd be required. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)
On Mon, 12 Feb 2024 at 15:32, slfan1989 wrote: > > > Note, because the arm64 binaries are built separately on a different > platform and JVM, their jar files may not match those of the x86 > release -and therefore the maven artifacts. I don't think this is > an issue (the ASF actually releases source tarballs, the binaries are > there for help only, though with the maven repo that's a bit blurred). > > The only way to be consistent would actually untar the x86.tar.gz, > overwrite its binaries with the arm stuff, retar, sign and push out > for the vote. that's exactly what the "arm.release" target in my client-validator does. builds an arm tar with the x86 binaries but the arm native libs, signs it. > Even automating that would be risky. > > automating is the *only* way to do it; apache ant has everything needed for this including the ability to run gpg. we did this on the relevant 3.3.x releases and nobody has yet complained...
[jira] [Resolved] (HADOOP-19057) S3 public test bucket landsat-pds unreadable -needs replacement
[
https://issues.apache.org/jira/browse/HADOOP-19057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19057.
-
Fix Version/s: 3.3.9
Resolution: Fixed
> S3 public test bucket landsat-pds unreadable -needs replacement
> ---
>
> Key: HADOOP-19057
> URL: https://issues.apache.org/jira/browse/HADOOP-19057
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0, 3.2.4, 3.3.9, 3.3.6, 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.3.9, 3.5.0, 3.4.1
>
>
> The s3 test bucket used in hadoop-aws tests of S3 select and large file reads
> is no longer publicly accessible
> {code}
> java.nio.file.AccessDeniedException: landsat-pds: getBucketMetadata() on
> landsat-pds: software.amazon.awssdk.services.s3.model.S3Exception: null
> (Service: S3, Status Code: 403, Request ID: 06QNYQ9GND5STQ2S, Extended
> Request ID:
> O+u2Y1MrCQuuSYGKRAWHj/5LcDLuaFS8owNuXXWSJ0zFXYfuCaTVLEP351S/umti558eKlUqV6U=):null
> {code}
> * Because HADOOP-18830 has cut s3 select, all we need in 3.4.1+ is a large
> file for some reading tests
> * changing the default value disables s3 select tests on older releases
> * if fs.s3a.scale.test.csvfile is set to " " then other tests which need it
> will be skipped
> Proposed
> * we locate a new large file under the (requester pays) s3a://usgs-landsat/
> bucket . All releases with HADOOP-18168 can use this
> * update 3.4.1 source to use this; document it
> * do something similar for 3.3.9 + maybe even cut s3 select there too.
> * document how to use it on older releases with requester-pays support
> * document how to completely disable it on older releases.
> h2. How to fix (most) landsat test failures on older releases
> add this to your auth-keys.xml file. Expect some failures in a few tests
> with-hardcoded references to the bucket (assumed role delegation tokens)
> {code}
>
> fs.s3a.scale.test.csvfile
> s3a://noaa-cors-pds/raw/2023/017/ohfh/OHFH017d.23_.gz
> file used in scale tests
>
>
> fs.s3a.bucket.noaa-cors-pds.endpoint.region
> us-east-1
>
>
> fs.s3a.bucket.noaa-isd-pds.multipart.purge
> false
> Don't try to purge uploads in the read-only bucket, as
> it will only create log noise.
>
>
> fs.s3a.bucket.noaa-isd-pds.probe
> 0
> Let's postpone existence checks to the first IO operation
>
>
>
> fs.s3a.bucket.noaa-isd-pds.audit.add.referrer.header
> false
> Do not add the referrer header
>
>
> fs.s3a.bucket.noaa-isd-pds.prefetch.block.size
> 128k
> Use a small prefetch size so tests fetch multiple
> blocks
>
>
> fs.s3a.select.enabled
> false
>
> {code}
> Some delegation token tests will still fail; these have hard-coded references
> to the old bucket. *Do not worry about these*
> {code}
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens[0] »
> AccessDenied s3a://la...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens[1] »
> AccessDenied s3a://la...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens[2] »
> AccessDenied s3a://la...
> [ERROR]
> ITestRoleDelegationInFilesystem>ITestSessionDelegationInFilesystem.testDelegatedFileSystem:347->ITestSessionDelegationInFilesystem.readLandsatMetadata:614
> » AccessDenied
> [ERROR]
> ITestSessionDelegationInFilesystem.testDelegatedFileSystem:347->readLandsatMetadata:614
> » AccessDenied
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: build q: getting the full DAG or all maven dependencies
it does, but i'm not sure if there is a single module where you can ask for it and get the full list. For that verification project I've got I may declare more poms as dependencies so can do the aggregate scan there. this would also let me run maven dependency -verbose, save the output to a file and see what is there. would let us define lists of libraries we don't want in distributions On Mon, 12 Feb 2024 at 18:03, Sangjin Lee wrote: > Does the maven dependency plugin help? I might try mvn dependency:tree and > see if it takes you somewhere. > > Sangjin > > > On Mon, Feb 12, 2024 at 9:50 AM Steve Loughran > > wrote: > > > how can we work out the entire DAG of dependencies in a hadoop distro? > > > > I'm asking as there are things in 3.4.0 that we shouldn't need (protobuf > > 2.5), and when I add the pR to move off log4j 1.17 to reload4j, I still > > find one in the yarn timeline lib dir > > https://github.com/apache/hadoop/pull/6547 > > > > > > see HADOOP-19074 for a list of what is in 3.4.0 RC0, which predates the > new > > shaded jar. > > >
[jira] [Resolved] (HADOOP-18930) S3A: make fs.s3a.create.performance an option you can set for the entire bucket
[ https://issues.apache.org/jira/browse/HADOOP-18930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18930. - Fix Version/s: 3.4.0 (was: 3.3.7-aws) Resolution: Fixed > S3A: make fs.s3a.create.performance an option you can set for the entire > bucket > --- > > Key: HADOOP-18930 > URL: https://issues.apache.org/jira/browse/HADOOP-18930 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.9 > Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > make the fs.s3a.create.performance option something you can set everywhere, > rather than just in an openFile() option or under a magic path. > this improves performance on apps like iceberg where filenames are generated > with UUIDs in them, so we know there are no overwrites -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
build q: getting the full DAG or all maven dependencies
how can we work out the entire DAG of dependencies in a hadoop distro? I'm asking as there are things in 3.4.0 that we shouldn't need (protobuf 2.5), and when I add the pR to move off log4j 1.17 to reload4j, I still find one in the yarn timeline lib dir https://github.com/apache/hadoop/pull/6547 see HADOOP-19074 for a list of what is in 3.4.0 RC0, which predates the new shaded jar.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)
so it'll have the s3a checksum and region stuff? that'd be wonderful! I've been assuming i'd have to push out a 3.4.1 release for that alone. On Sat, 10 Feb 2024 at 23:36, slfan1989 wrote: > We will end the voting for hadoop-3.4.0-RC1 and will open the voting for > hadoop-3.4.0-RC2 in 1-2 days. > > * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly > submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0. > * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0. > > Best Regards, > Shilun Fan. > > On Mon, Jan 29, 2024 at 10:33 PM slfan1989 wrote: > > > Apache Hadoop 3.4.0 > > > > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop > > 3.4.0. > > > > What we would like is for anyone who can to verify the tarballs, > especially > > anyone who can try the arm64 binaries as we want to include them too. > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/ > > > > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1 > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1395/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md > > > > Release notes > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md > > > > This is off branch-3.4.0 and is the first big release since 3.3.6. > > > > Key changes include > > > > * S3A: Upgrade AWS SDK to V2 > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks > > * YARN Federation improvements > > * YARN Capacity Scheduler improvements > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes > > * HDFS EC: Code Enhancements and Bug Fixes > > * Transitive CVE fixes > > > > Differences from RC0 > > > > * We've improved Hadoop 3.4.0 Highlight big features and improvements. > > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules. > > * Use validate-hadoop-client-artifacts[1] for packaging and verification. > > > > Note, because the arm64 binaries are built separately on a different > > platform and JVM, their jar files may not match those of the x86 > > release -and therefore the maven artifacts. I don't think this is > > an issue (the ASF actually releases source tarballs, the binaries are > > there for help only, though with the maven repo that's a bit blurred). > > > > The only way to be consistent would actually untar the x86.tar.gz, > > overwrite its binaries with the arm stuff, retar, sign and push out > > for the vote. Even automating that would be risky. > > > > [1] validate-hadoop-client-artifacts: > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > Thanks to steve for providing validate-hadoop-client-artifacts. > > > > Best Regards, > > Shilun Fan. > > >
[jira] [Resolved] (HADOOP-19059) update AWS SDK to support S3 Access Grants in S3A
[ https://issues.apache.org/jira/browse/HADOOP-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19059. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > update AWS SDK to support S3 Access Grants in S3A > - > > Key: HADOOP-19059 > URL: https://issues.apache.org/jira/browse/HADOOP-19059 > Project: Hadoop Common > Issue Type: Improvement > Components: build, fs/s3 >Affects Versions: 3.4.0 >Reporter: Jason Han >Assignee: Jason Han >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > Original Estimate: 168h > Remaining Estimate: 168h > > In order to support S3 Access > Grants(https://aws.amazon.com/s3/features/access-grants/) in S3A, we need to > update AWS SDK in hadooop package. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19072) S3A: expand optimisations on stores with "fs.s3a.create.performance"
Steve Loughran created HADOOP-19072: --- Summary: S3A: expand optimisations on stores with "fs.s3a.create.performance" Key: HADOOP-19072 URL: https://issues.apache.org/jira/browse/HADOOP-19072 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran on an s3a store with fs.s3a.create.performance set, speed up other operations * mkdir to skip parent directory check: just do a HEAD to see if there's a file at the target location -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19045) S3A: pass request timeouts down to sdk clients
[ https://issues.apache.org/jira/browse/HADOOP-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19045. - Resolution: Fixed > S3A: pass request timeouts down to sdk clients > -- > > Key: HADOOP-19045 > URL: https://issues.apache.org/jira/browse/HADOOP-19045 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > s3a client timeout settings are getting down to http client, but not sdk > timeouts, so you can't have a longer timeout than the default. This surfaces > in the inability to tune the timeouts for CreateSession calls even now the > latest SDK does pick it up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-19045) S3A: pass request timeouts down to sdk clients
[ https://issues.apache.org/jira/browse/HADOOP-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19045: - this is broken because core-default.xml sets it to 0 > S3A: pass request timeouts down to sdk clients > -- > > Key: HADOOP-19045 > URL: https://issues.apache.org/jira/browse/HADOOP-19045 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > s3a client timeout settings are getting down to http client, but not sdk > timeouts, so you can't have a longer timeout than the default. This surfaces > in the inability to tune the timeouts for CreateSession calls even now the > latest SDK does pick it up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18993) Allow to not isolate S3AFileSystem classloader when needed
[
https://issues.apache.org/jira/browse/HADOOP-18993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18993.
-
Fix Version/s: 3.5.0
Resolution: Fixed
> Allow to not isolate S3AFileSystem classloader when needed
> --
>
> Key: HADOOP-18993
> URL: https://issues.apache.org/jira/browse/HADOOP-18993
> Project: Hadoop Common
> Issue Type: Improvement
> Components: hadoop-thirdparty
>Affects Versions: 3.3.6
>Reporter: Antonio Murgia
>Assignee: Antonio Murgia
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be
> the same as the one that loaded S3AFileSystem. This leads to the
> impossibility in Spark applications to load third party credentials providers
> as user jars.
> I propose to add a configuration key
> {{fs.s3a.extensions.isolated.classloader}} with a default value of {{true}}
> that if set to {{false}} will not perform the classloader set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0
I'd like to get a 3.3.x out with the release too, so as to end the emails we get to security@ listing everything someone's security scanner has found and demanding a timeline for a fix. Actually I should get back to the last such reporter and ask them to test the new RC and 3.4.x on the basis that they will be expected to upgrade, and now is the chance to identify any problems On Wed, 31 Jan 2024 at 20:13, PJ Fanning wrote: > +1 (non-binding) > > * I validated the checksum and signature on the src tgz > * LICENSE/NOTICE present > * ASF headers > * no unexpected binaries > * can build using mvn > * tested the thirdparty protobuf jar in hadoop main build > > Is the idea that there will be a Hadoop 3.4.0 RC2 that uses the thirdparty > jars after they are released? > > > On 2024/01/31 02:16:47 slfan1989 wrote: > > Thank you for the review and vote! Looking forward to other forks helping > > with voting and verification. > > > > Best Regards, > > Shilun Fan. > > > > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He > wrote: > > > > > Thanks Shilun for driving it and making it happen. > > > > > > +1(binding). > > > > > > [x] Checksums and PGP signatures are valid. > > > [x] LICENSE files exist. > > > [x] NOTICE is included. > > > [x] Rat check is ok. `mvn clean apache-rat:check` > > > [x] Built from source works well: `mvn clean install` > > > [x] Built Hadoop trunk with updated thirdparty successfully (include > update > > > protobuf shaded path). > > > > > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we > > > could finish this vote before 2024/02/06(UTC) if there are no concerns. > > > Thanks all. > > > > > > Best Regards, > > > - He Xiaoqiao > > > > > > > > > > > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 > wrote: > > > > > > > Hi folks, > > > > > > > > Xiaoqiao He and I have put together a release candidate (RC0) for > Hadoop > > > > Thirdparty 1.2.0. > > > > > > > > The RC is available at: > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0 > > > > > > > > The RC tag is > > > > > > > > https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0 > > > > > > > > The maven artifacts are staged at > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1398 > > > > > > > > Comparing to 1.1.1, there are three additional fixes: > > > > > > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12 > > > > https://github.com/apache/hadoop-thirdparty/pull/26 > > > > > > > > HADOOP-18921. Upgrade to avro 1.11.3 > > > > https://github.com/apache/hadoop-thirdparty/pull/24 > > > > > > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 > > > > https://github.com/apache/hadoop-thirdparty/pull/23 > > > > > > > > You can find my public key at : > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Best Regards, > > > > Shilun Fan. > > > > > > > > > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
[jira] [Resolved] (HADOOP-19049) Class loader leak caused by StatisticsDataReferenceCleaner thread
[ https://issues.apache.org/jira/browse/HADOOP-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19049. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > Class loader leak caused by StatisticsDataReferenceCleaner thread > - > > Key: HADOOP-19049 > URL: https://issues.apache.org/jira/browse/HADOOP-19049 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.6 >Reporter: Jia Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > The > "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" > daemon thread was created by FileSystem. > This is fine if the thread's context class loader is the system class loader, > but it's bad if the context class loader is a custom class loader. The > reference held by this daemon thread means that the class loader can never > become eligible for GC. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19044) AWS SDK V2 - Update S3A region logic
[ https://issues.apache.org/jira/browse/HADOOP-19044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19044. - Fix Version/s: 3.5.0 Resolution: Fixed > AWS SDK V2 - Update S3A region logic > - > > Key: HADOOP-19044 > URL: https://issues.apache.org/jira/browse/HADOOP-19044 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set > fs.s3a.endpoint to > s3.amazonaws.com here: > [https://github.com/apache/spark/blob/9a2f39318e3af8b3817dc5e4baf52e548d82063c/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L540] > > > HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is > set, or if a region can be parsed from fs.s3a.endpoint (which will happen in > this case, region will be US_EAST_1), cross region access is not enabled. > This will cause 400 errors if the bucket is not in US_EAST_1. > > Proposed: Updated the logic so that if the endpoint is the global > s3.amazonaws.com , cross region access is enabled. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18987) Corrections to Hadoop FileSystem API Definition
[ https://issues.apache.org/jira/browse/HADOOP-18987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18987. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > Corrections to Hadoop FileSystem API Definition > --- > > Key: HADOOP-18987 > URL: https://issues.apache.org/jira/browse/HADOOP-18987 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.6 >Reporter: Dieter De Paepe >Assignee: Dieter De Paepe >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > I noticed a lot of inconsistencies, typos and informal statements in the > "formal" FileSystem API definition > ([https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/index.html)] > Creating this ticket to link my PR against. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17784) hadoop-aws landsat-pds test bucket will be deleted after Jul 1, 2021
[ https://issues.apache.org/jira/browse/HADOOP-17784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17784. - Resolution: Duplicate HADOOP-17784 will address this now the bucket is completely gone > hadoop-aws landsat-pds test bucket will be deleted after Jul 1, 2021 > > > Key: HADOOP-17784 > URL: https://issues.apache.org/jira/browse/HADOOP-17784 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3, test >Reporter: Leona Yoda >Priority: Major > Attachments: org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob.txt > > > I found an anouncement that landsat-pds buket will be deleted on July 1, 2021 > (https://registry.opendata.aws/landsat-8/) > and I think this bucket is used in th test of hadoop-aws module use > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java#L93] > > At this time I can access the bucket but we might have to change the test > bucket in someday. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19057) S3 public test bucket landsat-pds unreadable -needs replacement
Steve Loughran created HADOOP-19057:
---
Summary: S3 public test bucket landsat-pds unreadable -needs
replacement
Key: HADOOP-19057
URL: https://issues.apache.org/jira/browse/HADOOP-19057
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.6, 3.2.4, 3.4.0, 3.3.9, 3.5.0
Reporter: Steve Loughran
The s3 test bucket used in hadoop-aws tests of S3 select and large file reads
is no longer publicly accessible
{code}
java.nio.file.AccessDeniedException: landsat-pds: getBucketMetadata() on
landsat-pds: software.amazon.awssdk.services.s3.model.S3Exception: null
(Service: S3, Status Code: 403, Request ID: 06QNYQ9GND5STQ2S, Extended Request
ID:
O+u2Y1MrCQuuSYGKRAWHj/5LcDLuaFS8owNuXXWSJ0zFXYfuCaTVLEP351S/umti558eKlUqV6U=):null
{code}
* Because HADOOP-18830 has cut s3 select, all we need in 3.4.1+ is a large file
for some reading tests
* changing the default value disables s3 select tests on older releases
* if fs.s3a.scale.test.csvfile is set to " " then other tests which need it
will be skipped
Proposed
* we locate a new large file under the (requester pays) s3a://usgs-landsat/
bucket . All releases with HADOOP-18168 can use this
* update 3.4.1 source to use this; document it
* do something similar for 3.3.9 + maybe even cut s3 select there too.
* document how to use it on older releases with requester-pays support
* document how to completely disable it on older releases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19022) S3A : ITestS3AConfiguration#testRequestTimeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19022.
-
Fix Version/s: 3.5.0
3.4.1
Assignee: Steve Loughran
Resolution: Duplicate
> S3A : ITestS3AConfiguration#testRequestTimeout failure
> --
>
> Key: HADOOP-19022
> URL: https://issues.apache.org/jira/browse/HADOOP-19022
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
> Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.5.0, 3.4.1
>
>
> "fs.s3a.connection.request.timeout" should be specified in milliseconds as per
> {code:java}
> Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
> DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO);
> {code}
> The test fails consistently because it sets 120 ms timeout which is less than
> 15s (min network operation duration), and hence gets reset to 15000 ms based
> on the enforcement.
>
> {code:java}
> [ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.016 s <<< FAILURE!
> java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is
> different than what AWS sdk configuration uses internally expected:<12>
> but was:<15000>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19045) S3A: pass request timeouts down to sdk clients
[ https://issues.apache.org/jira/browse/HADOOP-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19045. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > S3A: pass request timeouts down to sdk clients > -- > > Key: HADOOP-19045 > URL: https://issues.apache.org/jira/browse/HADOOP-19045 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > s3a client timeout settings are getting down to http client, but not sdk > timeouts, so you can't have a longer timeout than the default. This surfaces > in the inability to tune the timeouts for CreateSession calls even now the > latest SDK does pick it up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18830) S3A: Cut S3 Select
[
https://issues.apache.org/jira/browse/HADOOP-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18830.
-
Fix Version/s: 3.4.1
Hadoop Flags: Incompatible change
Release Note: S3 Select is no longer supported through the S3A connector
Resolution: Fixed
> S3A: Cut S3 Select
> --
>
> Key: HADOOP-18830
> URL: https://issues.apache.org/jira/browse/HADOOP-18830
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1
>
>
> getting s3 select to work with the v2 sdk is tricky, we need to add extra
> libraries to the classpath beyond just bundle.jar. we can do this but
> * AFAIK nobody has ever done CSV predicate pushdown, as it breaks split logic
> completely
> * CSV is a bad format
> * one-line JSON more structured but also way less efficient
> ORC/Parquet benefit from vectored IO and work spanning the cluster.
> accordingly, I'm wondering what to do about s3 select
> # cut?
> # downgrade to optional and document the extra classes on the classpath
> Option #2 is straightforward and effectively the default. we can also declare
> the feature deprecated.
> {code}
> [ERROR]
> testReadLandsatRecordsNoMatch(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat)
> Time elapsed: 147.958 s <<< ERROR!
> java.io.IOException: java.lang.NoClassDefFoundError:
> software/amazon/eventstream/MessageDecoder
> at
> org.apache.hadoop.fs.s3a.select.SelectObjectContentHelper.select(SelectObjectContentHelper.java:75)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$select$10(WriteOperationHelper.java:660)
> at
> org.apache.hadoop.fs.store.audit.AuditingFunctions.lambda$withinAuditSpan$0(AuditingFunctions.java:62)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19046) S3A: update sdk versions
[
https://issues.apache.org/jira/browse/HADOOP-19046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19046.
-
Fix Version/s: 3.5.0
Resolution: Fixed
> S3A: update sdk versions
>
>
> Key: HADOOP-19046
> URL: https://issues.apache.org/jira/browse/HADOOP-19046
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Move up to the most recent versions of the v2 sdk, with a v1 update just to
> keep some CVE checking happy.
> {code}
> 1.12.599
> 2.23.5
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[ https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18975. - Resolution: Fixed > AWS SDK v2: extend support for FIPS endpoints > -- > > Key: HADOOP-18975 > URL: https://issues.apache.org/jira/browse/HADOOP-18975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > v1 SDK supported FIPS just by changing the endpoint. > Now we have a new builder setting to use. > * add new fs.s3a.endpoint.fips option > * pass it down > * test -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19015) Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool
[ https://issues.apache.org/jira/browse/HADOOP-19015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19015. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed > Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting > for connection from pool > -- > > Key: HADOOP-19015 > URL: https://issues.apache.org/jira/browse/HADOOP-19015 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > Getting errors in jobs which can be fixed by increasing this > 2023-12-14 17:35:56,602 [ERROR] [TezChild] |tez.TezProcessor|: > java.lang.RuntimeException: java.io.IOException: > org.apache.hadoop.net.ConnectTimeoutException: getFileStatus on > s3a://aaa/cc-hive-jzv5y6/warehouse/tablespace/managed/hive/student/delete_delta_012_012_0001/bucket_1_0: > software.amazon.awssdk.core.exception.SdkClientException: Unable to execute > HTTP request: Timeout waiting for connection from pool at > org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206) > at > org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) > at > org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:84) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:70) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:70) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:40) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptible -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18883. - Fix Version/s: 3.5.0 Resolution: Fixed > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19048) ItestCustomSigner failing against S3Express Buckets
Steve Loughran created HADOOP-19048:
---
Summary: ItestCustomSigner failing against S3Express Buckets
Key: HADOOP-19048
URL: https://issues.apache.org/jira/browse/HADOOP-19048
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.5.0
Reporter: Steve Loughran
getting test failures against S3 Express buckets with {{ItestCustomSigner}};
not seen with classic s3 stores.
{code}
[ERROR]
testCustomSignerAndInitializer[simple-delete](org.apache.hadoop.fs.s3a.auth.ITestCustomSigner)
Time elapsed: 6.12 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT 0-byte object on
fork-0006/test/testCustomSignerAndInitializer[simple-delete]/customsignerpath1:
software.amazon.awssdk.services.s3.model.S3Exception:
x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-*
or x-amz-trailer headers were found. (Service: S3, Status Code: 400, Request
ID: 0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID:
kZJZG05LGCBu7lsNKNf):InvalidRequest: x-amz-sdk-checksum-algorithm specified,
but no corresponding x-amz-checksum-* or x-amz-trailer headers were found.
(Service: S3, Status Code: 400, Request ID:
0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID:
kZJZG05LGCBu7lsNKNf)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:259)
...
Caused by: software.amazon.awssdk.services.s3.model.S3Exception:
x-amz-sdk-checksum-algorithm specified, but no corresponding x-amz-checksum-*
or x-amz-trailer headers were found. (Service: S3, Status Code: 400, Request
ID: 0033eada6b00018d21962f1b05094a80435cca52, Extended Request ID:
kZJZG05LGCBu7lsNKNf)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19033) S3A: disable checksum validation
[ https://issues.apache.org/jira/browse/HADOOP-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19033. - Fix Version/s: 3.5.0 Resolution: Fixed > S3A: disable checksum validation > > > Key: HADOOP-19033 > URL: https://issues.apache.org/jira/browse/HADOOP-19033 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 > Reporter: Steve Loughran > Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > AWS v2 sdk turns on client-side checksum validation; this kills performance > Given we are using TLS to download from AWS s3, there's implicit channel > checksumming going on on, that's along with the IPv4 TCP checksumming. > We don't need it, all it does is slow us down. > proposed: disable in DefaultS3ClientFactory > I don't want to add an option to enable it as it only complicates life (yet > another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17824) ITestCustomSigner fails with NPE against private endpoint
[ https://issues.apache.org/jira/browse/HADOOP-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17824. - Resolution: Cannot Reproduce stack trace no longer valid for v2 sdk; closing as cannot reproduce > ITestCustomSigner fails with NPE against private endpoint > - > > Key: HADOOP-17824 > URL: https://issues.apache.org/jira/browse/HADOOP-17824 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.1 > Reporter: Steve Loughran >Priority: Minor > > ITestCustomSigner fails when the tester is pointed at a private endpoint -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19046) S3A: update sdk versions
Steve Loughran created HADOOP-19046:
---
Summary: S3A: update sdk versions
Key: HADOOP-19046
URL: https://issues.apache.org/jira/browse/HADOOP-19046
Project: Hadoop Common
Issue Type: Sub-task
Components: build, fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Move up to the most recent versions of the v2 sdk, with a v1 update just to
keep some CVE checking happy.
{code}
1.12.599
2.23.5
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19045) S3A: pass request timeouts down to sdk clients
Steve Loughran created HADOOP-19045: --- Summary: S3A: pass request timeouts down to sdk clients Key: HADOOP-19045 URL: https://issues.apache.org/jira/browse/HADOOP-19045 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran s3a client timeout settings are getting down to http client, but not sdk timeouts, so you can't have a longer timeout than the default. This surfaces in the inability to tune the timeouts for CreateSession calls even now the latest SDK does pick it up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
Steve Loughran created HADOOP-19043:
---
Summary: S3A: Regression: ITestS3AOpenCost fails on prefetch test
runs
Key: HADOOP-19043
URL: https://issues.apache.org/jira/browse/HADOOP-19043
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Getting test failures in the new ITestS3AOpenCost tests when run with
{{-Dprefetch}}
Thought I'd tested this, but clearly not
* class cast failures on asserts (fix: skip)
* bytes read different in one test: (fix: identify and address)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19042) S3A: detect and recover from SSL ConnectionReset exceptions
Steve Loughran created HADOOP-19042: --- Summary: S3A: detect and recover from SSL ConnectionReset exceptions Key: HADOOP-19042 URL: https://issues.apache.org/jira/browse/HADOOP-19042 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.6, 3.4.0 Reporter: Steve Loughran s3a input stream doesn't recover from SSL exceptions, specifically ConnectionReset This is a variant of HADOOP-19027, except it's surfaced on an older release... # need to make sure the specific exception is handled by aborting stream and retrying -so map to the new HttpChannelEOFException # all of thisd needs to be backported -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19027.
-
Fix Version/s: 3.5.0
Resolution: Fixed
in 3.5 though I hope to backport to 3.4.1
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Fw:Re: [VOTE] Release Apache Hadoop 3.4.0 RC0
-1 I'm afraid, just due to staging/packaging issues.
This took me a few goes to get right myself, so nothing unusual.
Note I used my validator project which is set to retrieve binaries, check
signatures, run maven builds against staged artifacts *and clean up any
local copies first*and more.
This uses apache ant to manage all this:
https://github.com/steveloughran/validate-hadoop-client-artifacts
Here's the initial build.properties:file I used to try and manage this
## build.properties:
hadoop.version=3.4.0
rc=RC0
amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/
http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64
release=hadoop-${hadoop.version}-RC0
rc.dirname=${release}
release.native.binaries=false
git.commit.id=cdb8af4f22ec
nexus.staging.url=
https://repository.apache.org/content/repositories/orgapachehadoop-1391/
hadoop.source.dir=${local.dev.dir}/hadoop-trunk
##
When I did my own builds, all the artifacts created were without the RC0
suffix. It is critical this happens because the .sha512 checksums include
that in their paths
> cat hadoop-3.4.0-RC0.tar.gz.sha512
SHA512 (hadoop-3.4.0-RC0.tar.gz) =
e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab
Maven artifacts: staging problems
Couldn't build with a -Pstaging profile as the staging repository wasn't
yet closed -I tried to do that myself.
This failed with some rule problem
Event: Failed: Checksum Validation
Monday, January 15, 2024 14:37:13 GMT (GMT+)
typeId checksum-staging
failureMessage INVALID SHA-1:
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1'
failureMessage Requires one-of SHA-1:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1,
SHA-256:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256,
SHA-512:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512
I don't know precisely what this means...my guess is that the upload didn't
include everything.
Note my client-validator module can check this; just run its maven test
commands
mvn clean test -U -P3.4 -Pstaging
GPG signing: all good.
Picked your key up from the site ( ant gpg.keys ) ... first validation with
ant gpg.verify was unhappy as your key wasn't trusted. I've signed it and
pushed that signature up, so people who trust me get some reassurance about
you.
My build then failed as the gpg code couldn't find the
hadoop-3.4.0-aarch64.tar.gz.asc
The problem here is that although we want separate arm and x86 tar files,
we don't really want separate binaries as it only creates different jars in
the wild.
The way I addressed that was after creating that x86 release on an ec2 vm
and downloading it, I then did a local arm64 build and then created an arm
.tar.gz file, copied it into the same dir as the amd66 binaries but with
the arm64 .tar.gz filename, .asc and .sha512 checksum files all renamed
(checksum file patches to match the name).
https://github.com/steveloughran/validate-hadoop-client-artifacts?tab=readme-ov-file#arm64-binaries
