[jira] [Resolved] (HADOOP-19244) Pullout arch-agnostic maven javadoc plugin configurations in hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-19244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19244. - Fix Version/s: 3.5.0 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Pullout arch-agnostic maven javadoc plugin configurations in hadoop-common > -- > > Key: HADOOP-19244 > URL: https://issues.apache.org/jira/browse/HADOOP-19244 > Project: Hadoop Common > Issue Type: Improvement > Components: build, common >Reporter: Cheng Pan >Assignee: Cheng Pan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19222) Switch yum repo baseurl due to CentOS 7 sunset
[ https://issues.apache.org/jira/browse/HADOOP-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19222. - Hadoop Flags: Reviewed Resolution: Fixed > Switch yum repo baseurl due to CentOS 7 sunset > -- > > Key: HADOOP-19222 > URL: https://issues.apache.org/jira/browse/HADOOP-19222 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0, 3.4.1 >Reporter: Cheng Pan >Assignee: Cheng Pan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > Similar to HADOOP-18151 (which handled sunset for CentOS 8), CentOS 7 reached > EOL on July 1, 2024 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19173) Upgrade org.apache.derby:derby to 10.17.1.0
Shilun Fan created HADOOP-19173: --- Summary: Upgrade org.apache.derby:derby to 10.17.1.0 Key: HADOOP-19173 URL: https://issues.apache.org/jira/browse/HADOOP-19173 Project: Hadoop Common Issue Type: Improvement Components: build, common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan Upgrade org.apache.derby:derby to 10.17.1.0. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19135) Remove Jcache 1.0-alpha
[ https://issues.apache.org/jira/browse/HADOOP-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19135. - Fix Version/s: 3.5.0 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Remove Jcache 1.0-alpha > --- > > Key: HADOOP-19135 > URL: https://issues.apache.org/jira/browse/HADOOP-19135 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.5.0, 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > In YARN Federation, we use JCache. The version of JCache has not been > maintained for a long time. We directly use ECache instead of JCache in > YARN-11663, so we can remove JCache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19143) Upgrade commons-cli to 1.6.0.
Shilun Fan created HADOOP-19143: --- Summary: Upgrade commons-cli to 1.6.0. Key: HADOOP-19143 URL: https://issues.apache.org/jira/browse/HADOOP-19143 Project: Hadoop Common Issue Type: Improvement Components: build, common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan commons-cli can be upgraded to 1.6.0, I will try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19136) Upgrade commons-io to 2.15.0
Shilun Fan created HADOOP-19136: --- Summary: Upgrade commons-io to 2.15.0 Key: HADOOP-19136 URL: https://issues.apache.org/jira/browse/HADOOP-19136 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan commons-io can be upgraded from 2.14.0 to 2.15.0, try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19135) Remove Jcache 1.0-alpha
Shilun Fan created HADOOP-19135: --- Summary: Remove Jcache 1.0-alpha Key: HADOOP-19135 URL: https://issues.apache.org/jira/browse/HADOOP-19135 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan In YARN Federation, we use JCache. The version of JCache has not been maintained for a long time. We directly use ECache instead of JCache in YARN-11663, so we can remove JCache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19052.
-
Fix Version/s: 3.5.0
3.4.1
Hadoop Flags: Reviewed
Target Version/s: 3.5.0, 3.4.1
Resolution: Fixed
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a new process, and will return the result
> in a very short time.
>
> When we use this method to get the file linkCount, we rarely get the WARN log
> above when the QPS of append execution is high.
> .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19124) Update org.ehcache from 3.3.1 to 3.8.2.
Shilun Fan created HADOOP-19124: --- Summary: Update org.ehcache from 3.3.1 to 3.8.2. Key: HADOOP-19124 URL: https://issues.apache.org/jira/browse/HADOOP-19124 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan We need to enhance the caching functionality in Yarn Federation by adding a limit on the number of cached entries. I noticed that the version of org.ehcache is relatively old and requires an upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19018.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19117) 3.4.0 release documents
[ https://issues.apache.org/jira/browse/HADOOP-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19117. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > 3.4.0 release documents > --- > > Key: HADOOP-19117 > URL: https://issues.apache.org/jira/browse/HADOOP-19117 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
[ https://issues.apache.org/jira/browse/HADOOP-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19112. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Hadoop 3.4.0 release wrap-up > > > Key: HADOOP-19112 > URL: https://issues.apache.org/jira/browse/HADOOP-19112 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19117) 3.4.0 release documents
Shilun Fan created HADOOP-19117: --- Summary: 3.4.0 release documents Key: HADOOP-19117 URL: https://issues.apache.org/jira/browse/HADOOP-19117 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
Shilun Fan created HADOOP-19112: --- Summary: Hadoop 3.4.0 release wrap-up Key: HADOOP-19112 URL: https://issues.apache.org/jira/browse/HADOOP-19112 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19099) Add Protobuf Compatibility Notes
[ https://issues.apache.org/jira/browse/HADOOP-19099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19099. - Fix Version/s: 3.4.0 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add Protobuf Compatibility Notes > > > Key: HADOOP-19099 > URL: https://issues.apache.org/jira/browse/HADOOP-19099 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > > In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version > 3.21.12. This version may have compatibility issues with certain versions of > JDK8. We will document this situation in the index.md file of hadoop-3.4.0 > and inform users that we will discontinue support for JDK8 in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19099) Add Protobuf Compatibility Notes
Shilun Fan created HADOOP-19099: --- Summary: Add Protobuf Compatibility Notes Key: HADOOP-19099 URL: https://issues.apache.org/jira/browse/HADOOP-19099 Project: Hadoop Common Issue Type: Sub-task Components: documentation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version 3.21.12. This version may have compatibility issues with certain versions of JDK8. We will document this situation in the index.md file of hadoop-3.4.0 and inform users that we will discontinue support for JDK8 in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19084) prune dependency exports of hadoop-* modules
[
https://issues.apache.org/jira/browse/HADOOP-19084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19084.
-
Hadoop Flags: Reviewed
Resolution: Fixed
> prune dependency exports of hadoop-* modules
>
>
> Key: HADOOP-19084
> URL: https://issues.apache.org/jira/browse/HADOOP-19084
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.4.0, 3.5.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> this is probably caused by HADOOP-18613:
> ZK is pulling in some extra transitive stuff which surfaces in applications
> which import hadoop-common into their poms. It doesn't seem to show up in our
> distro, but downstream you get warnings about duplicate logging stuff
> {code}
> | +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
> | | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
> | | | \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile -
> omitted for duplicate)
> | | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
> | | +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for
> conflict with 4.1.100.Final)
> | | +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile -
> omitted for conflict with 4.1.100.Final)
> | | +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
> | | +- ch.qos.logback:logback-core:jar:1.2.10:compile
> | | +- ch.qos.logback:logback-classic:jar:1.2.10:compile
> | | | +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for
> duplicate)
> | | | \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict
> with 1.7.30)
> | | \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict
> with 2.14.0)
> {code}
> proposed: exclude the zk dependencies we either override outselves or don't
> need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19075) add shaded jersey jar hadoop-shaded-jersey
Shilun Fan created HADOOP-19075: --- Summary: add shaded jersey jar hadoop-shaded-jersey Key: HADOOP-19075 URL: https://issues.apache.org/jira/browse/HADOOP-19075 Project: Hadoop Common Issue Type: Improvement Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan We need to address the issue of upgrading Jersey, replacing Jersey 1.x with Jersey 2.x. Jersey 2 and Jersey 1 are completely different, with many differences, and the same project cannot simultaneously introduce Jersey 2 and Jersey 1. In order to allow the project to reference both Jersey 2 and Jersey 1 simultaneously, facilitating the step-by-step module upgrade process, I plan to introduce Jersey 2 into hadoop-thirdparty. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19069) Use hadoop-thirdparty 1.2.0
[ https://issues.apache.org/jira/browse/HADOOP-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19069. - Fix Version/s: 3.4.0 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Use hadoop-thirdparty 1.2.0 > --- > > Key: HADOOP-19069 > URL: https://issues.apache.org/jira/browse/HADOOP-19069 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: 3.4.0, 3.5.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19071) Update maven-surefire-plugin from 3.0.0 to 3.2.2
Shilun Fan created HADOOP-19071: --- Summary: Update maven-surefire-plugin from 3.0.0 to 3.2.2 Key: HADOOP-19071 URL: https://issues.apache.org/jira/browse/HADOOP-19071 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.5.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19070) Java 17 compile support
Shilun Fan created HADOOP-19070: --- Summary: Java 17 compile support Key: HADOOP-19070 URL: https://issues.apache.org/jira/browse/HADOOP-19070 Project: Hadoop Common Issue Type: Task Reporter: Shilun Fan We plan to make Hadoop compatible with Java 17. There is a lot of work to be done, but we still aim to complete this task by 2024. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19053) Release Hadoop-Thirdparty 1.2.0
[ https://issues.apache.org/jira/browse/HADOOP-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19053. - Fix Version/s: thirdparty-1.3.0 thirdparty-1.2.0 Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0, thirdparty-1.3.0 Resolution: Fixed > Release Hadoop-Thirdparty 1.2.0 > --- > > Key: HADOOP-19053 > URL: https://issues.apache.org/jira/browse/HADOOP-19053 > Project: Hadoop Common > Issue Type: Task >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Fix For: thirdparty-1.3.0, thirdparty-1.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19069) Use hadoop-thirdparty 1.2.0
Shilun Fan created HADOOP-19069: --- Summary: Use hadoop-thirdparty 1.2.0 Key: HADOOP-19069 URL: https://issues.apache.org/jira/browse/HADOOP-19069 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0, 3.5.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19063) [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy
[ https://issues.apache.org/jira/browse/HADOOP-19063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19063. - Fix Version/s: thirdparty-1.3.0 Hadoop Flags: Reviewed Resolution: Fixed > [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy > - > > Key: HADOOP-19063 > URL: https://issues.apache.org/jira/browse/HADOOP-19063 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.3.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.3.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > setuptools-scm and lazy-object-proxy have been upgraded many times. Directly > relying on pip to install dependencies will report errors. We need to specify > the installation version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19062) Improve create-release RUN script
[
https://issues.apache.org/jira/browse/HADOOP-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19062.
-
Fix Version/s: thirdparty-1.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Improve create-release RUN script
> -
>
> Key: HADOOP-19062
> URL: https://issues.apache.org/jira/browse/HADOOP-19062
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: hadoop-thirdparty
>Affects Versions: thirdparty-1.3.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: thirdparty-1.3.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Using create-release will create a docker image locally, but three of the RUN
> scripts may fail to execute.
> 1. RUN groupadd --non-unique -g 0 root
> {code:java}
> => ERROR [16/20] RUN groupadd --non-unique -g 0 root
> 0.2s
> --
> > [16/20] RUN groupadd --non-unique -g 0 root:
> 0.154 groupadd: group 'root' already exists
> --
> Dockerfile:100
>
> 98 |
> 99 | LABEL org.apache.hadoop.create-release="cr-19697"
> 100 | >>> RUN groupadd --non-unique -g 0 root
> 101 | RUN useradd -g 0 -u 0 -m root
> 102 | RUN chown -R root /home/root
> {code}
> 2. RUN useradd -g 0 -u 0 -m root
> {code:java}
> > [17/20] RUN useradd -g 0 -u 0 -m root:
> 0.165 useradd: user 'root' already exists
> --
> Dockerfile:101
>
> 99 | LABEL org.apache.hadoop.create-release="cr-12068"
> 100 | RUN groupadd --non-unique -g 0 root; exit 0;
> 101 | >>> RUN useradd -g 0 -u 0 -m root
> 102 | RUN chown -R root /home/root
> 103 | ENV HOME /home/root
> {code}
> 3. RUN chown -R root /home/root
> {code:java}
> > [18/20] RUN chown -R root /home/root:
> 0.168 chown: cannot access '/home/root': No such file or directory
> --
> Dockerfile:102
>
> 100 | RUN groupadd --non-unique -g 0 root; exit 0;
> 101 | RUN useradd -g 0 -u 0 -m root; exit 0;
> 102 | >>> RUN chown -R root /home/root
> 103 | ENV HOME /home/root
> 104 | RUN mkdir -p /maven
>
> {code}
> Even if these three scripts fail, subsequent steps can continue to be
> executed, so I added exit 0 after the script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19064) [thirdparty] add -mvnargs option to create-release command line
[ https://issues.apache.org/jira/browse/HADOOP-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19064. - Fix Version/s: thirdparty-1.3.0 Hadoop Flags: Reviewed Resolution: Fixed > [thirdparty] add -mvnargs option to create-release command line > > > Key: HADOOP-19064 > URL: https://issues.apache.org/jira/browse/HADOOP-19064 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.3.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.3.0 > > Time Spent: 40m > Remaining Estimate: 0h > > We need mvn to support extended parameters. This JIRA is similar to > HADOOP-18198. > I use this parameter to refer to the user.home. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19064) [thirdparty] add -mvnargs option to create-release command line
Shilun Fan created HADOOP-19064: --- Summary: [thirdparty] add -mvnargs option to create-release command line Key: HADOOP-19064 URL: https://issues.apache.org/jira/browse/HADOOP-19064 Project: Hadoop Common Issue Type: Sub-task Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan We need mvn to support extended parameters. This JIRA is similar to HADOOP-18198. I use this parameter to refer to the user.home. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19063) [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy
Shilun Fan created HADOOP-19063: --- Summary: [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy Key: HADOOP-19063 URL: https://issues.apache.org/jira/browse/HADOOP-19063 Project: Hadoop Common Issue Type: Sub-task Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan setuptools-scm and lazy-object-proxy have been upgraded many times. Directly relying on pip to install dependencies will report errors. We need to specify the installation version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19062) Improve create-release RUN script
Shilun Fan created HADOOP-19062:
---
Summary: Improve create-release RUN script
Key: HADOOP-19062
URL: https://issues.apache.org/jira/browse/HADOOP-19062
Project: Hadoop Common
Issue Type: Sub-task
Components: hadoop-thirdparty
Affects Versions: 3.5.0
Reporter: Shilun Fan
Assignee: Shilun Fan
Using create-release will create a docker image locally, but three of the RUN
scripts may fail to execute.
1. RUN groupadd --non-unique -g 0 root
{code:java}
=> ERROR [16/20] RUN groupadd --non-unique -g 0 root
0.2s
--
> [16/20] RUN groupadd --non-unique -g 0 root:
0.154 groupadd: group 'root' already exists
--
Dockerfile:100
98 |
99 | LABEL org.apache.hadoop.create-release="cr-19697"
100 | >>> RUN groupadd --non-unique -g 0 root
101 | RUN useradd -g 0 -u 0 -m root
102 | RUN chown -R root /home/root
{code}
2. RUN useradd -g 0 -u 0 -m root
{code:java}
> [17/20] RUN useradd -g 0 -u 0 -m root:
0.165 useradd: user 'root' already exists
--
Dockerfile:101
99 | LABEL org.apache.hadoop.create-release="cr-12068"
100 | RUN groupadd --non-unique -g 0 root; exit 0;
101 | >>> RUN useradd -g 0 -u 0 -m root
102 | RUN chown -R root /home/root
103 | ENV HOME /home/root
{code}
3. RUN chown -R root /home/root
{code:java}
> [18/20] RUN chown -R root /home/root:
0.168 chown: cannot access '/home/root': No such file or directory
--
Dockerfile:102
100 | RUN groupadd --non-unique -g 0 root; exit 0;
101 | RUN useradd -g 0 -u 0 -m root; exit 0;
102 | >>> RUN chown -R root /home/root
103 | ENV HOME /home/root
104 | RUN mkdir -p /maven
{code}
Even if these three scripts fail, subsequent steps can continue to be executed,
so I added exit 0 after the script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19055) Preparing for 1.3.0 development
[ https://issues.apache.org/jira/browse/HADOOP-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19055. - Fix Version/s: thirdparty-1.2.0 Resolution: Fixed > Preparing for 1.3.0 development > --- > > Key: HADOOP-19055 > URL: https://issues.apache.org/jira/browse/HADOOP-19055 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.3.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0, thirdparty-1.3.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19054) Update hadoop-thirdparty index.md.vm
[ https://issues.apache.org/jira/browse/HADOOP-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19054. - Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0, thirdparty-1.3.0 (was: thirdparty-1.2.0) Resolution: Fixed > Update hadoop-thirdparty index.md.vm > > > Key: HADOOP-19054 > URL: https://issues.apache.org/jira/browse/HADOOP-19054 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19055) Preparing for 1.3.0 development
Shilun Fan created HADOOP-19055: --- Summary: Preparing for 1.3.0 development Key: HADOOP-19055 URL: https://issues.apache.org/jira/browse/HADOOP-19055 Project: Hadoop Common Issue Type: Sub-task Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix For: thirdparty-1.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19054) Update hadoop-thirdparty index.md.vm
Shilun Fan created HADOOP-19054: --- Summary: Update hadoop-thirdparty index.md.vm Key: HADOOP-19054 URL: https://issues.apache.org/jira/browse/HADOOP-19054 Project: Hadoop Common Issue Type: Sub-task Affects Versions: thirdparty-1.2.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix For: thirdparty-1.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18197. - Fix Version/s: thirdparty-1.2.0 Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 Resolution: Fixed > Update protobuf 3.7.1 to a version without CVE-2021-22569 > - > > Key: HADOOP-18197 > URL: https://issues.apache.org/jira/browse/HADOOP-18197 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ivan Viaznikov >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available, security > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The artifact `org.apache.hadoop:hadoop-common` brings in a dependency > `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version > released in 2013 and it contains a vulnerability > [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. > Therefore, requesting you to clarify if this library version is going to be > updated in the following releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18342) Upgrade to Avro 1.11.1
[ https://issues.apache.org/jira/browse/HADOOP-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18342. - Fix Version/s: thirdparty-1.2.0 Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 Resolution: Fixed > Upgrade to Avro 1.11.1 > -- > > Key: HADOOP-18342 > URL: https://issues.apache.org/jira/browse/HADOOP-18342 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Latest version of Avro. Aimed only at trunk as there is no security concern > addressed here. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19053) Release Hadoop-Thirdparty 1.2.0
Shilun Fan created HADOOP-19053: --- Summary: Release Hadoop-Thirdparty 1.2.0 Key: HADOOP-19053 URL: https://issues.apache.org/jira/browse/HADOOP-19053 Project: Hadoop Common Issue Type: Task Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19039) Hadoop 3.4.0 Highlight big features and improvements.
[ https://issues.apache.org/jira/browse/HADOOP-19039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19039. - Fix Version/s: 3.4.0 3.5.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0, 3.5.0, 3.4.1 Resolution: Fixed > Hadoop 3.4.0 Highlight big features and improvements. > - > > Key: HADOOP-19039 > URL: https://issues.apache.org/jira/browse/HADOOP-19039 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > > While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion > of numerous commits in this version. Therefore, highlighting significant > features and improvements becomes crucial. I've completed the initial > version and now seek the review of more experienced partner to ensure the > finalization of the version's highlights. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18959) Use builder for prefetch CachingBlockManager
[ https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18959. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use builder for prefetch CachingBlockManager > > > Key: HADOOP-18959 > URL: https://issues.apache.org/jira/browse/HADOOP-18959 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Some of the recent changes (HADOOP-18399, HADOOP-18291, HADOOP-18829 etc) > have added more params for prefetch CachingBlockManager c'tor to process > read/write block requests. They have added too many params and more are > likely to be introduced later. We should use builder pattern to pass params. > This would also help consolidating required prefetch params into one single > place within S3ACachingInputStream, from scattered locations. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18894) upgrade sshd-core due to CVEs
[ https://issues.apache.org/jira/browse/HADOOP-18894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18894. - Fix Version/s: 3.4.0 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade sshd-core due to CVEs > - > > Key: HADOOP-18894 > URL: https://issues.apache.org/jira/browse/HADOOP-18894 > Project: Hadoop Common > Issue Type: Bug >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > > https://mvnrepository.com/artifact/org.apache.sshd/sshd-core > hadoop currently uses v1.6.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19038) Improve create-release RUN script
[
https://issues.apache.org/jira/browse/HADOOP-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19038.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Target Version/s: 3.4.0 (was: 3.5.0)
Resolution: Fixed
> Improve create-release RUN script
> -
>
> Key: HADOOP-19038
> URL: https://issues.apache.org/jira/browse/HADOOP-19038
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.4.0, 3.5.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Using create-release will create a docker image locally, but three of the RUN
> scripts may fail to execute.
> 1. RUN groupadd --non-unique -g 0 root
> {code:java}
> => ERROR [16/20] RUN groupadd --non-unique -g 0 root
> 0.2s
> --
> > [16/20] RUN groupadd --non-unique -g 0 root:
> 0.154 groupadd: group 'root' already exists
> --
> Dockerfile:100
>
> 98 |
> 99 | LABEL org.apache.hadoop.create-release="cr-19697"
> 100 | >>> RUN groupadd --non-unique -g 0 root
> 101 | RUN useradd -g 0 -u 0 -m root
> 102 | RUN chown -R root /home/root
> {code}
> 2. RUN useradd -g 0 -u 0 -m root
> {code:java}
> > [17/20] RUN useradd -g 0 -u 0 -m root:
> 0.165 useradd: user 'root' already exists
> --
> Dockerfile:101
>
> 99 | LABEL org.apache.hadoop.create-release="cr-12068"
> 100 | RUN groupadd --non-unique -g 0 root; exit 0;
> 101 | >>> RUN useradd -g 0 -u 0 -m root
> 102 | RUN chown -R root /home/root
> 103 | ENV HOME /home/root
> {code}
> 3. RUN chown -R root /home/root
> {code:java}
> > [18/20] RUN chown -R root /home/root:
> 0.168 chown: cannot access '/home/root': No such file or directory
> --
> Dockerfile:102
>
> 100 | RUN groupadd --non-unique -g 0 root; exit 0;
> 101 | RUN useradd -g 0 -u 0 -m root; exit 0;
> 102 | >>> RUN chown -R root /home/root
> 103 | ENV HOME /home/root
> 104 | RUN mkdir -p /maven
>
> {code}
> Even if these three scripts fail, subsequent steps can continue to be
> executed, so I added exit 0 after the script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19040) mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes.
[ https://issues.apache.org/jira/browse/HADOOP-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19040. - Hadoop Flags: Reviewed Resolution: Fixed > mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes. > --- > > Key: HADOOP-19040 > URL: https://issues.apache.org/jira/browse/HADOOP-19040 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18895) upgrade to commons-compress 1.24.0 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-18895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18895. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade to commons-compress 1.24.0 due to CVE > - > > Key: HADOOP-18895 > URL: https://issues.apache.org/jira/browse/HADOOP-18895 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Includes some important bug fixes including > https://lists.apache.org/thread/g9lrsz8j9nrgltcoc7v6cpkopg07czc9 - > CVE-2023-42503 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18865) ABFS: Adding 100 continue in userAgent String and dynamically removing it if retry is without the header enabled.
[ https://issues.apache.org/jira/browse/HADOOP-18865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18865. - Hadoop Flags: Reviewed Resolution: Fixed > ABFS: Adding 100 continue in userAgent String and dynamically removing it if > retry is without the header enabled. > - > > Key: HADOOP-18865 > URL: https://issues.apache.org/jira/browse/HADOOP-18865 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Adding 100 continue in userAgent String if enabled in AbfsConfiguration and > dynamically removing it if retry is without the header enabled. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19040) mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes.
Shilun Fan created HADOOP-19040: --- Summary: mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes. Key: HADOOP-19040 URL: https://issues.apache.org/jira/browse/HADOOP-19040 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19039) Hadoop 3.4.0 Highlight big features and improvements.
Shilun Fan created HADOOP-19039: --- Summary: Hadoop 3.4.0 Highlight big features and improvements. Key: HADOOP-19039 URL: https://issues.apache.org/jira/browse/HADOOP-19039 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion of numerous commits in this version. Therefore, highlighting significant features and improvements becomes crucial. I've completed the initial version and now seek the review of more experienced partner to ensure the finalization of the version's highlights. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19038) improve create-release RUN script
Shilun Fan created HADOOP-19038: --- Summary: improve create-release RUN script Key: HADOOP-19038 URL: https://issues.apache.org/jira/browse/HADOOP-19038 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0, 3.5.0 Reporter: Shilun Fan Assignee: Shilun Fan Using create-release will create a docker image locally, but three of the RUN scripts may fail to execute. 1. RUN groupadd --non-unique -g 0 root If the group already exists, this script will report an error. 2. RUN useradd -g 0 -u 0 -m root 3. RUN chown -R root /home/root -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19036) Standardize Maven Initialization Across Operating Systems
Shilun Fan created HADOOP-19036: --- Summary: Standardize Maven Initialization Across Operating Systems Key: HADOOP-19036 URL: https://issues.apache.org/jira/browse/HADOOP-19036 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0, 3.5.0 Reporter: Shilun Fan Assignee: Shilun Fan The differences in initializing Maven for various operating systems in the build scripts are as follows: - For Ubuntu and Debian, Maven is installed using the yum repository. - For CentOS 7 and CentOS 8, Maven is downloaded remotely. - The Maven version used on Windows is inconsistent with other operating systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19034) Fix Download Maven Url Not Found
Shilun Fan created HADOOP-19034: --- Summary: Fix Download Maven Url Not Found Key: HADOOP-19034 URL: https://issues.apache.org/jira/browse/HADOOP-19034 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In the process of preparing Hadoop 3.4.0 Release, I found that when opening the link to download maven, it will prompt not found. We need to fix this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18843. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19018) Release 3.4.0
Shilun Fan created HADOOP-19018: --- Summary: Release 3.4.0 Key: HADOOP-19018 URL: https://issues.apache.org/jira/browse/HADOOP-19018 Project: Hadoop Common Issue Type: Task Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Confirmed features to be included in the release: - Enhanced functionality for YARN Federation. - Optimization of HDFS RBF. - Introduction of fine-grained global locks for DataNodes. - Improvements in the stability of HDFS EC, and more. - Fixes for important CVEs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18880) Add some rpc related metrics to Metrics.md
[ https://issues.apache.org/jira/browse/HADOOP-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18880. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add some rpc related metrics to Metrics.md > -- > > Key: HADOOP-18880 > URL: https://issues.apache.org/jira/browse/HADOOP-18880 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Fix For: 3.4.0 > > > Add some rpc related metrics to Metrics.md > such as > # DeferredRpcProcessingTime > # RpcResponseTime > # RpcRequeueCalls -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18370) Fix missing package-info in hadoop-common moudle
[ https://issues.apache.org/jira/browse/HADOOP-18370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18370. - Resolution: Won't Fix > Fix missing package-info in hadoop-common moudle > > > Key: HADOOP-18370 > URL: https://issues.apache.org/jira/browse/HADOOP-18370 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.4 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Attachments: checkStyle Error.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > When reading the code, I found that some packages are missing package-info, > fix this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18810. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: common, documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are many configs in core-site.xml, which should > be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
[
https://issues.apache.org/jira/browse/HADOOP-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-18836.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Some properties are missing from hadoop-policy.xml.
> ---
>
> Key: HADOOP-18836
> URL: https://issues.apache.org/jira/browse/HADOOP-18836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, documentation, security
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
>
> {noformat}
> security.datanode.lifeline.protocol.acl
> security.get.user.mappings.protocol.acl
> security.reconfiguration.protocol.acl
> security.refresh.callqueue.protocol.acl
> security.refresh.generic.protocol.acl{noformat}
> The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18779) Improve hadoop-function.sh#status script
Shilun Fan created HADOOP-18779: --- Summary: Improve hadoop-function.sh#status script Key: HADOOP-18779 URL: https://issues.apache.org/jira/browse/HADOOP-18779 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan When we use yarn --daemon status router/nodemanager, I found that no information can be displayed. I checked the script and modified the script of hadoop-functions.sh#status to increase the output information. yarn --daemon status router - If the service is started router is running as process 47626. - If the service is stop router is stopped. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18485) Upgrade hikari.version from 4.0.3 to 5.0.1
[ https://issues.apache.org/jira/browse/HADOOP-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18485. - Resolution: Won't Fix > Upgrade hikari.version from 4.0.3 to 5.0.1 > -- > > Key: HADOOP-18485 > URL: https://issues.apache.org/jira/browse/HADOOP-18485 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > Upgrade hikari.version from 4.0.3 to 5.0.1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18595) Fix "the the" and friends typos
[ https://issues.apache.org/jira/browse/HADOOP-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18595. - Resolution: Fixed > Fix "the the" and friends typos > --- > > Key: HADOOP-18595 > URL: https://issues.apache.org/jira/browse/HADOOP-18595 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Priority: Major > Fix For: 3.4.0 > > > Fix "the the" and friends typos #5267 Contributed by @neshkeev -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18595) Fix "the the" and friends typos
Shilun Fan created HADOOP-18595: --- Summary: Fix "the the" and friends typos Key: HADOOP-18595 URL: https://issues.apache.org/jira/browse/HADOOP-18595 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Fix "the the" and friends typos #5267 Contributed by @neshkeev -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
