[jira] [Created] (HADOOP-18298) Hadoop AWS | Staging committer Multipartupload not implemented properly
Ayush Goyal created HADOOP-18298:
Summary: Hadoop AWS | Staging committer Multipartupload not
implemented properly
Key: HADOOP-18298
URL: https://issues.apache.org/jira/browse/HADOOP-18298
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.1
Reporter: Ayush Goyal
In Hadoop aws staging
committer(org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter), Committer
uploads files from local to s3(method- commitTaskInternal) which calls
uploadFileToPendingCommit of CommitOperation to upload file using multipart
upload.
Multipart upload consists of three steps-
1)Initialise multipartupload.
2) Breaks the file to part and upload Parts.
3) Merge all the parts of files and finalize multipart.
In the implementation of uploadFileToPendingCommit, first 2 steps are
implemented. However, 3rd part is missing which leads to uploading the parts
file but because it is not merged at the end of job no files are there in
destination directory.
S3 logs before implement 3rd steps-
{code:java}
2022-05-30T13:49:31:000 [200 OK] s3.NewMultipartUpload
localhost:9000/minio-feature-testing/spark-job/processed/output-parquet-staging-7/part-0-ce0a965f-622a-4950-bb4b-550470883134-c000-b552fb34-6156-4aa8-9085-679ad14fab6e.snappy.parquet?uploads
240b:c1d1:123:664f:c5d2:2:: 8.677ms ↑ 137 B ↓ 724 B
2022-05-30T13:49:31:000 [200 OK] s3.PutObjectPart
localhost:9000/minio-feature-testing/spark-job/processed/output-parquet-staging-7/part-0-ce0a965f-622a-4950-bb4b-550470883134-c000-b552fb34-6156-4aa8-9085-679ad14fab6e.snappy.parquet?uploadId=f3beae8e-3001-48be-9bc4-306b71940e50=1
240b:c1d1:123:664f:c5d2:2:: 443.156ms ↑ 51 KiB ↓ 325 B
2022-05-30T13:49:32:000 [200 OK] s3.ListObjectsV2
localhost:9000/minio-feature-testing/?list-type=2=%2F=2=spark-job%2Fprocessed%2Foutput-parquet-staging-7%2F_SUCCESS%2F=false
240b:c1d1:123:664f:c5d2:2:: 3.414ms ↑ 137 B ↓ 646 B
2022-05-30T13:49:32:000 [200 OK] s3.PutObject
localhost:9000/minio-feature-testing/spark-job/processed/output-parquet-staging-7/_SUCCESS
240b:c1d1:123:664f:c5d2:2:: 52.734ms ↑ 8.7 KiB ↓ 380 B
2022-05-30T13:49:32:000 [200 OK] s3.DeleteMultipleObjects
localhost:9000/minio-feature-testing/?delete 240b:c1d1:123:664f:c5d2:2::
73.954ms ↑ 350 B ↓ 432 B
2022-05-30T13:49:32:000 [404 Not Found] s3.HeadObject
localhost:9000/minio-feature-testing/spark-job/processed/output-parquet-staging-7/_temporary
240b:c1d1:123:664f:c5d2:2:: 2.658ms ↑ 137 B ↓ 291 B
2022-05-30T13:49:32:000 [200 OK] s3.ListObjectsV2
localhost:9000/minio-feature-testing/?list-type=2=%2F=2=spark-job%2Fprocessed%2Foutput-parquet-staging-7%2F_temporary%2F=false
240b:c1d1:123:664f:c5d2:2:: 4.807ms ↑ 137 B ↓ 648 B
2022-05-30T13:49:32:000 [200 OK] s3.ListMultipartUploads
localhost:9000/minio-feature-testing/?uploads=spark-job%2Fprocessed%2Foutput-parquet-staging-7%2F
240b:c0e0:102:553e:b4c2:2:: 1.081ms ↑ 137 B ↓ 776 B
2022-05-30T13:49:32:000 [404 Not Found] s3.HeadObject
localhost:9000/minio-feature-testing/spark-job/processed/output-parquet-staging-7/.spark-staging-ce0a965f-622a-4950-bb4b-550470883134
240b:c1d1:123:664f:c5d2:2:: 5.68ms ↑ 137 B ↓ 291 B
2022-05-30T13:49:32:000 [200 OK] s3.ListObjectsV2
localhost:9000/minio-feature-testing/?list-type=2=%2F=2=spark-job%2Fprocessed%2Foutput-parquet-staging-7%2F.spark-staging-ce0a965f-622a-4950-bb4b-550470883134%2F=false
240b:c1d1:123:664f:c5d2:2:: 2.452ms ↑ 137 B ↓ 689 B
{code}
Here , After s3.PutObjectPart there is no completeMultipartupload call for 3rd
step.
S3 logs after implement 3rd steps-
{code:java}
2022-06-17T10:56:12:000 [200 OK] s3.NewMultipartUpload
localhost:9000/minio-feature-testing/spark-job/pm-processed/output-parquet-staging-39/day%3D23/hour%3D16/quarter%3D0/part-4-d0b529ca-112f-43f2-a7dd-44de4db6aa7f-dffa7213-d492-48f9-9e6a-fb08bc81ceeb.c000.snappy.parquet?uploads
240b:c1d1:123:664f:c5d2:2:: 9.116ms ↑ 137 B ↓ 750 B
2022-06-17T10:56:12:000 [200 OK] s3.NewMultipartUpload
localhost:9000/minio-feature-testing/spark-job/pm-processed/output-parquet-staging-39/day%3D23/hour%3D15/quarter%3D45/part-4-d0b529ca-112f-43f2-a7dd-44de4db6aa7f-dffa7213-d492-48f9-9e6a-fb08bc81ceeb.c000.snappy.parquet?uploads
240b:c1d1:123:664f:c5d2:2:: 9.416ms ↑ 137 B ↓ 751 B
2022-06-17T10:56:12:000 [200 OK] s3.NewMultipartUpload
localhost:9000/minio-feature-testing/spark-job/pm-processed/output-parquet-staging-39/day%3D23/hour%3D16/quarter%3D45/part-4-d0b529ca-112f-43f2-a7dd-44de4db6aa7f-dffa7213-d492-48f9-9e6a-fb08bc81ceeb.c000.snappy.parquet?uploads
240b:c1d1:123:664f:c5d2:2:: 8.506ms ↑ 137 B ↓ 751 B
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/ [Jun 9, 2022 1:42:04 PM] (Akira Ajisaka) HDFS-16576. Remove unused imports in HDFS project (#4389) [Jun 10, 2022 4:03:25 PM] (Steve Loughran) YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi (#4417) [Jun 10, 2022 7:02:47 PM] (Chris Nauroth) HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.yarn.server.resourcemanager.TestClientRMTokens hadoop.yarn.client.api.impl.TestAMRMClient hadoop.mapred.TestLocalDistributedCacheManager hadoop.mapreduce.security.TestJHSSecurity cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-compile-javac-root.txt [388K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [60K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [168K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [52K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt [32K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/60/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [96K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18297) Upgrade dependency-check-maven to 7.1.1
Ashutosh Gupta created HADOOP-18297: --- Summary: Upgrade dependency-check-maven to 7.1.1 Key: HADOOP-18297 URL: https://issues.apache.org/jira/browse/HADOOP-18297 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta The OWASP dependency-check-maven Plugin version has corrected various false positives in 7.1.1. We can upgrade to it. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18289) Remove WhiteBox in hadoop-kms module.
[ https://issues.apache.org/jira/browse/HADOOP-18289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18289. -- Fix Version/s: 3.4.0 Resolution: Fixed Done. Thanks [~slfan1989] > Remove WhiteBox in hadoop-kms module. > - > > Key: HADOOP-18289 > URL: https://issues.apache.org/jira/browse/HADOOP-18289 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > WhiteBox is deprecated, try to remove this method in hadoop-kms. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/ [Jun 15, 2022 9:58:10 AM] (noreply) HDFS-16469. Locate protoc-gen-hrpc across platforms (#4434) [Jun 15, 2022 9:59:55 AM] (noreply) YARN-11078. Set env vars in a cross platform compatible way (#4432) [Jun 15, 2022 1:16:24 PM] (noreply) HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. [Jun 15, 2022 7:03:26 PM] (noreply) HADOOP-18159. Bump cos_api-bundle to 5.6.69 to update public-suffix-list.txt (#) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.yarn.server.resourcemanager.TestClientRMTokens hadoop.mapred.TestLocalDistributedCacheManager hadoop.mapreduce.security.TestJHSSecurity cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/results-javadoc-javadoc-root.txt [400K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [568K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [188K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt [48K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/893/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [100K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.
Mukund Thakur created HADOOP-18296: -- Summary: Memory fragmentation in ChecksumFileSystem Vectored IO implementation. Key: HADOOP-18296 URL: https://issues.apache.org/jira/browse/HADOOP-18296 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Mukund Thakur As we have implemented merging of ranges in the ChecksumFSInputChecker implementation of vectored IO api, it can lead to memory fragmentation. Let me explain by example. Suppose client requests for 3 ranges. 0-500, 700-1000 and 1200-1500. Now because of merging, all the above ranges will get merged into one and we will allocate a big byte buffer of 0-1500 size but return sliced byte buffers for the desired ranges. Now once the client is done reading all the ranges, it will only be able to free the memory for requested ranges and memory of the gaps will never be released for eg here (500-700 and 1000-1200). Note this only happens for direct byte buffers. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18107) Vectored IO support for large S3 files.
[ https://issues.apache.org/jira/browse/HADOOP-18107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukund Thakur resolved HADOOP-18107. Resolution: Fixed Merged in feature branch [https://github.com/apache/hadoop/commits/feature-vectored-io] > Vectored IO support for large S3 files. > > > Key: HADOOP-18107 > URL: https://issues.apache.org/jira/browse/HADOOP-18107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > This effort would mostly be adding more tests for large files under scale > tests and see if any new issue surfaces. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18105) Implement a variant of ElasticByteBufferPool which uses weak references for garbage collection.
[ https://issues.apache.org/jira/browse/HADOOP-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukund Thakur resolved HADOOP-18105. Resolution: Fixed Merged in feature branch [https://github.com/apache/hadoop/commits/feature-vectored-io] > Implement a variant of ElasticByteBufferPool which uses weak references for > garbage collection. > --- > > Key: HADOOP-18105 > URL: https://issues.apache.org/jira/browse/HADOOP-18105 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, fs >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > Currently in hadoop codebase, we have two classes which implements byte > buffers pooling. > One is ElasticByteBufferPool which doesn't use weak references and thus could > cause memory leaks in production environment. > Other is DirectBufferPool which uses weak references but doesn't support > caller's preference for either on-heap or off-heap buffers. > > The idea is to create an improved version of ElasticByteBufferPool by > subclassing it ( as it is marked as public and stable and used widely in hdfs > ) with essential functionalities required for effective buffer pooling. This > is important for the parent Vectored IO work. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18295) Add S3A configuration property for `no_proxy` hosts
Sam Kramer created HADOOP-18295: --- Summary: Add S3A configuration property for `no_proxy` hosts Key: HADOOP-18295 URL: https://issues.apache.org/jira/browse/HADOOP-18295 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Sam Kramer Seeing as there are configuration options for proxy host, port, username, and password, there should also be an option to be able to provide to the S3 client a list of hosts to not use the proxy for (i.e. `no_proxy`) I'm happy to contribute the code, but figured I'd file a ticket first to see if this the hadoop community would be open to this idea or have any desire for this feature. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18294) Ensure build folder exists before writing checksum file.ProtocRunner#writeChecksums
Ashutosh Gupta created HADOOP-18294: --- Summary: Ensure build folder exists before writing checksum file.ProtocRunner#writeChecksums Key: HADOOP-18294 URL: https://issues.apache.org/jira/browse/HADOOP-18294 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Ensure build folder exists before writing checksum file.ProtocRunner#writeChecksums -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Where is aarch64 tgz for 3.3.2 and 3.3.3
> > I'm assuming that the arm docker image can do the build, either on my > macbook or more likely an arm linux vm in someones cloud infrastructure? Yes, it would be the same release script but should be run on an ARM machine. The create-release script handles it here: https://github.com/apache/hadoop/blob/1b25851ad9ca5d1fc12b67089bb5adaa53641245/dev-support/bin/create-release#L207-L210 if that's true, how do i get its artifacts into the release? is the idea I > just upload a whole new binary .tar.gz file with a suffix to make clear its > arm, while the rest of release (source, maven artifacts etc) comes from the > main one? Uploading the .tar.gz, the signature and the checksum files suffixed with -aarch64.tar.gz and so should do. Can check this commit, just upload files in a way that they get recognized by this code: https://github.com/apache/hadoop-site/commit/7487c909e835c10f6fe24aeb52d9784c4f06bdea for example if 3.3.3 had arm release the paths would have looked the way here: https://github.com/apache/hadoop-site/blob/asf-site/content/releases.html#L214-L216 -Ayush On Thu, 16 Jun 2022 at 15:03, Steve Loughran wrote: > i'm going to cut a 3.3.4 release this week, i will try to do an arm binary > too. > > on that topic: how do i do this? > > I'm assuming that the arm docker image can do the build, either on my > macbook or more likely an arm linux vm in someones cloud infrastructure? > > if that's true, how do i get its artifacts into the release? is the idea I > just upload a whole new binary .tar.gz file with a suffix to make clear its > arm, while the rest of release (source, maven artifacts etc) comes from the > main one? > > On Thu, 16 Jun 2022 at 09:56, Ayush Saxena wrote: > >> Same question here: >> >> https://lists.apache.org/thread/rjn4zhdc30zrorsyr8jcsovmbl5hq6ol >> >> -Ayush >> >> Sent from my iPhone >> >> > On 16-Jun-2022, at 10:11 AM, Cheng Pan wrote: >> > >> > Hi community, >> > >> > I'm using Hadoop 3.3.1 to build cross-platform Docker images, and when >> > I'm trying to upgrade to 3.3.2 or 3.3.3, I found there is no aarch64 >> > tgz like 3.3.1[1]. Where can I get them? >> > >> > [1] >> https://dlcdn.apache.org/hadoop/core/hadoop-3.3.1/hadoop-3.3.1-aarch64.tar.gz >> > >> > Thanks, >> > Cheng Pan >> > >> > - >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >> >
[jira] [Created] (HADOOP-18293) Release Hadoop 3.3.4 critical fix update
Steve Loughran created HADOOP-18293: --- Summary: Release Hadoop 3.3.4 critical fix update Key: HADOOP-18293 URL: https://issues.apache.org/jira/browse/HADOOP-18293 Project: Hadoop Common Issue Type: Task Components: build Reporter: Steve Loughran Assignee: Steve Loughran Create a new release off the branch-3.3.3 line with a few more changes * wrap up of security changes * cut hadoop-cos out of hadoop-cloud-storage as its dependencies break s3a client...reinstate once the updated jar is tested -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Where is aarch64 tgz for 3.3.2 and 3.3.3
i'm going to cut a 3.3.4 release this week, i will try to do an arm binary too. on that topic: how do i do this? I'm assuming that the arm docker image can do the build, either on my macbook or more likely an arm linux vm in someones cloud infrastructure? if that's true, how do i get its artifacts into the release? is the idea I just upload a whole new binary .tar.gz file with a suffix to make clear its arm, while the rest of release (source, maven artifacts etc) comes from the main one? On Thu, 16 Jun 2022 at 09:56, Ayush Saxena wrote: > Same question here: > > https://lists.apache.org/thread/rjn4zhdc30zrorsyr8jcsovmbl5hq6ol > > -Ayush > > Sent from my iPhone > > > On 16-Jun-2022, at 10:11 AM, Cheng Pan wrote: > > > > Hi community, > > > > I'm using Hadoop 3.3.1 to build cross-platform Docker images, and when > > I'm trying to upgrade to 3.3.2 or 3.3.3, I found there is no aarch64 > > tgz like 3.3.1[1]. Where can I get them? > > > > [1] > https://dlcdn.apache.org/hadoop/core/hadoop-3.3.1/hadoop-3.3.1-aarch64.tar.gz > > > > Thanks, > > Cheng Pan > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > >
Re: Where is aarch64 tgz for 3.3.2 and 3.3.3
Same question here: https://lists.apache.org/thread/rjn4zhdc30zrorsyr8jcsovmbl5hq6ol -Ayush Sent from my iPhone > On 16-Jun-2022, at 10:11 AM, Cheng Pan wrote: > > Hi community, > > I'm using Hadoop 3.3.1 to build cross-platform Docker images, and when > I'm trying to upgrade to 3.3.2 or 3.3.3, I found there is no aarch64 > tgz like 3.3.1[1]. Where can I get them? > > [1] > https://dlcdn.apache.org/hadoop/core/hadoop-3.3.1/hadoop-3.3.1-aarch64.tar.gz > > Thanks, > Cheng Pan > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >
