[jira] [Created] (HADOOP-19173) Upgrade org.apache.derby:derby to 10.17.1.0
Shilun Fan created HADOOP-19173: --- Summary: Upgrade org.apache.derby:derby to 10.17.1.0 Key: HADOOP-19173 URL: https://issues.apache.org/jira/browse/HADOOP-19173 Project: Hadoop Common Issue Type: Improvement Components: build, common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan Upgrade org.apache.derby:derby to 10.17.1.0. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19136) Upgrade commons-io to 2.16.1
[ https://issues.apache.org/jira/browse/HADOOP-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19136: Summary: Upgrade commons-io to 2.16.1 (was: Upgrade commons-io to 2.16.0) > Upgrade commons-io to 2.16.1 > > > Key: HADOOP-19136 > URL: https://issues.apache.org/jira/browse/HADOOP-19136 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > commons-io can be upgraded from 2.14.0 to 2.16.0, try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19135) Remove Jcache 1.0-alpha
[ https://issues.apache.org/jira/browse/HADOOP-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19135. - Fix Version/s: 3.5.0 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Remove Jcache 1.0-alpha > --- > > Key: HADOOP-19135 > URL: https://issues.apache.org/jira/browse/HADOOP-19135 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.5.0, 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > In YARN Federation, we use JCache. The version of JCache has not been > maintained for a long time. We directly use ECache instead of JCache in > YARN-11663, so we can remove JCache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19136) Upgrade commons-io to 2.16.0
[ https://issues.apache.org/jira/browse/HADOOP-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19136: Description: commons-io can be upgraded from 2.14.0 to 2.16.0, try to upgrade. (was: commons-io can be upgraded from 2.14.0 to 2.15.0, try to upgrade.) > Upgrade commons-io to 2.16.0 > > > Key: HADOOP-19136 > URL: https://issues.apache.org/jira/browse/HADOOP-19136 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > commons-io can be upgraded from 2.14.0 to 2.16.0, try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19143) Upgrade commons-cli to 1.6.0.
Shilun Fan created HADOOP-19143: --- Summary: Upgrade commons-cli to 1.6.0. Key: HADOOP-19143 URL: https://issues.apache.org/jira/browse/HADOOP-19143 Project: Hadoop Common Issue Type: Improvement Components: build, common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan commons-cli can be upgraded to 1.6.0, I will try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19136) Upgrade commons-io to 2.16.0
[ https://issues.apache.org/jira/browse/HADOOP-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19136: Summary: Upgrade commons-io to 2.16.0 (was: Upgrade commons-io to 2.15.0) > Upgrade commons-io to 2.16.0 > > > Key: HADOOP-19136 > URL: https://issues.apache.org/jira/browse/HADOOP-19136 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > > commons-io can be upgraded from 2.14.0 to 2.15.0, try to upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19124) Update org.ehcache from 3.3.1 to 3.8.2.
[ https://issues.apache.org/jira/browse/HADOOP-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833766#comment-17833766 ] Shilun Fan commented on HADOOP-19124: - [~ste...@apache.org] Thanks for reminding! I will cherrypick this pr to branch-3.4. > Update org.ehcache from 3.3.1 to 3.8.2. > --- > > Key: HADOOP-19124 > URL: https://issues.apache.org/jira/browse/HADOOP-19124 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.1 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > We need to enhance the caching functionality in Yarn Federation by adding a > limit on the number of cached entries. I noticed that the version of > org.ehcache is relatively old and requires an upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19135) Remove Jcache 1.0-alpha
Shilun Fan created HADOOP-19135: --- Summary: Remove Jcache 1.0-alpha Key: HADOOP-19135 URL: https://issues.apache.org/jira/browse/HADOOP-19135 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.5.0, 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan In YARN Federation, we use JCache. The version of JCache has not been maintained for a long time. We directly use ECache instead of JCache in YARN-11663, so we can remove JCache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19132) Upgrade hadoop3 docker scripts to 3.4.0
[ https://issues.apache.org/jira/browse/HADOOP-19132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832056#comment-17832056 ] Shilun Fan edited comment on HADOOP-19132 at 3/29/24 6:50 AM: -- [~adoroszlai] Thank you very much for suggesting the upgrade! Personally, I believe we should upgrade to the hadoop-3.4.1 version instead of hadoop-3.4.0. The reason is that hadoop-3.4.0 is not a stable version, and it seems that there are still improvements needed, especially protobuf. I suggest releasing a new Docker image based on hadoop-3.4.1. cc: [~hexiaoqiao] was (Author: slfan1989): [~adoroszlai] Thank you very much for suggesting the upgrade! Personally, I believe we should upgrade to the hadoop-3.4.1 version instead of hadoop-3.4.0. The reason is that hadoop-3.4.0 is not a stable version, and it seems that there are still improvements needed, especially protobuf. > Upgrade hadoop3 docker scripts to 3.4.0 > --- > > Key: HADOOP-19132 > URL: https://issues.apache.org/jira/browse/HADOOP-19132 > Project: Hadoop Common > Issue Type: Task >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-19052:
Component/s: fs
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.4.0
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a new process, and will return the result
> in a very short time.
>
> When we use this method to get the file linkCount, we rarely get the WARN log
> above when the QPS of append execution is high.
> .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-19052:
Affects Version/s: 3.4.0
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.4.0
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a new process, and will return the result
> in a very short time.
>
> When we use this method to get the file linkCount, we rarely get the WARN log
> above when the QPS of append execution is high.
> .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19052) Hadoop use Shell command to get the count of the hard link which takes a lot of time
[
https://issues.apache.org/jira/browse/HADOOP-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19052.
-
Fix Version/s: 3.5.0
3.4.1
Hadoop Flags: Reviewed
Target Version/s: 3.5.0, 3.4.1
Resolution: Fixed
> Hadoop use Shell command to get the count of the hard link which takes a lot
> of time
>
>
> Key: HADOOP-19052
> URL: https://issues.apache.org/jira/browse/HADOOP-19052
> Project: Hadoop Common
> Issue Type: Improvement
> Environment: Hadopp 3.3.4
>Reporter: liang yu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
> Attachments: debuglog.png
>
>
> Using Hadoop 3.3.4
>
> When the QPS of `append` executions is very high, at a rate of above 1/s.
>
> We found that the write speed in hadoop is very slow. We traced some
> datanodes' log and find that there is a warning :
> {code:java}
> 2024-01-26 11:09:44,292 WARN impl.FsDatasetImpl
> (InstrumentedLock.java:logwaitWarning(165)) Waited above threshold(300 ms) to
> acquire lock: lock identifier: FsDatasetRwlock waitTimeMs=336 ms.Suppressed 0
> lock wait warnings.Longest supressed waitTimeMs=0.The stack trace is
> java.lang.Thread,getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060)
> org.apache.hadoop.util.Instrumentedlock.logWaitWarning(InstrumentedLock.java:171)
> org.apache.hadoop.util.InstrumentedLock.check(InstrumentedLock.java:222)
> org.apache.hadoop.util.InstrumentedLock.lock(InstrumentedLock, iaya:105)
> org.apache.hadoop.util.AutocloseableLock.acquire(AutocloseableLock.java:67)
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.java:1239)
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:230)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver
> (DataXceiver.java:1313)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock
> (DataXceiver.java:764)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:176)
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:110)
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> Then we traced the method
> _org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.append(FsDatasetImpl.
> java:1239),_ and print how long each command take to finish the execution,
> and find that it takes us 700ms to get the linkCount of the file which is
> really slow.
> !debuglog.png!
>
> We traced the code and find that java1.8 use a Shell Command to get the
> linkCount, in which execution it will start a new Process and wait for the
> Process to fork, when the QPS is very high, it will sometimes take a long
> time to fork the process.
> Here is the shell command.
> {code:java}
> stat -c%h /path/to/file
> {code}
>
> Solution:
> For the FileStore that supports the file attributes "unix", we can use the
> method _Files.getAttribute(f.toPath(), "unix:nlink")_ to get the linkCount,
> this method doesn't need to start a new process, and will return the result
> in a very short time.
>
> When we use this method to get the file linkCount, we rarely get the WARN log
> above when the QPS of append execution is high.
> .
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-19123) update commons-configuration2 to 2.10.1 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-19123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-19123: --- Assignee: PJ Fanning > update commons-configuration2 to 2.10.1 due to CVE > -- > > Key: HADOOP-19123 > URL: https://issues.apache.org/jira/browse/HADOOP-19123 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > https://github.com/advisories/GHSA-9w38-p64v-xpmv -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19124) Update org.ehcache from 3.3.1 to 3.8.2.
Shilun Fan created HADOOP-19124: --- Summary: Update org.ehcache from 3.3.1 to 3.8.2. Key: HADOOP-19124 URL: https://issues.apache.org/jira/browse/HADOOP-19124 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.1 Reporter: Shilun Fan Assignee: Shilun Fan We need to enhance the caching functionality in Yarn Federation by adding a limit on the number of cached entries. I noticed that the version of org.ehcache is relatively old and requires an upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829565#comment-17829565 ] Shilun Fan commented on HADOOP-15984: - I will continue to follow up on HADOOP-15984. > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18487) Make protobuf 2.5 an optional runtime dependency.
[ https://issues.apache.org/jira/browse/HADOOP-18487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18487: Fix Version/s: (was: 3.4.0) > Make protobuf 2.5 an optional runtime dependency. > - > > Key: HADOOP-18487 > URL: https://issues.apache.org/jira/browse/HADOOP-18487 > Project: Hadoop Common > Issue Type: Improvement > Components: build, ipc >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > uses of protobuf 2.5 and RpcEnginej have been deprecated since 3.3.0 in > HADOOP-17046 > while still keeping those files around (for a long time...), how about we > make the protobuf 2.5.0 export off hadoop common and hadoop-hdfs *provided*, > rather than *compile* > that way, if apps want it for their own apis, they have to explicitly ask for > it, but at least our own scans don't break. > i have no idea what will happen to the rest of the stack at this point, it > will be "interesting" to see -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19089) [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path
[ https://issues.apache.org/jira/browse/HADOOP-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19089: Fix Version/s: (was: 3.4.0) > [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path > --- > > Key: HADOOP-19089 > URL: https://issues.apache.org/jira/browse/HADOOP-19089 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0, 3.4.1 >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > Fix For: 3.4.1 > > > A while back changes were made to support HDFS.setXAttr() and HDFS.getXAttr() > on root path for ABFS Driver. > For these, filesystem level APIs were introduced and used to set/get metadata > of container. > Refer to Jira: [HADOOP-18869] ABFS: Fixing Behavior of a File System APIs on > root path - ASF JIRA (apache.org) > Ideally, same set of APIs should be used, and root should be treated as a > path like any other path. > This change is to avoid calling container APIs for these HDFS calls. > As a result of this these APIs will fail on root path (as earlier) because > service does not support get/set of user properties on root path. > This change will also update the documentation to reflect that these > operations are not supported on root path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19018.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
[ https://issues.apache.org/jira/browse/HADOOP-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19112. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Hadoop 3.4.0 release wrap-up > > > Key: HADOOP-19112 > URL: https://issues.apache.org/jira/browse/HADOOP-19112 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19117) 3.4.0 release documents
[ https://issues.apache.org/jira/browse/HADOOP-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19117. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > 3.4.0 release documents > --- > > Key: HADOOP-19117 > URL: https://issues.apache.org/jira/browse/HADOOP-19117 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19117) 3.4.0 release documents
[ https://issues.apache.org/jira/browse/HADOOP-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19117: Target Version/s: 3.4.0 > 3.4.0 release documents > --- > > Key: HADOOP-19117 > URL: https://issues.apache.org/jira/browse/HADOOP-19117 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19117) 3.4.0 release documents
Shilun Fan created HADOOP-19117: --- Summary: 3.4.0 release documents Key: HADOOP-19117 URL: https://issues.apache.org/jira/browse/HADOOP-19117 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
[ https://issues.apache.org/jira/browse/HADOOP-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-19112: --- Assignee: Shilun Fan > Hadoop 3.4.0 release wrap-up > > > Key: HADOOP-19112 > URL: https://issues.apache.org/jira/browse/HADOOP-19112 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19112) Hadoop 3.4.0 release wrap-up
Shilun Fan created HADOOP-19112: --- Summary: Hadoop 3.4.0 release wrap-up Key: HADOOP-19112 URL: https://issues.apache.org/jira/browse/HADOOP-19112 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.4.0 Reporter: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824239#comment-17824239 ] Shilun Fan commented on HADOOP-15984: - I have removed the git bot (it will be added back later). I will be making frequent changes to this PR to avoid distracting everyone with compilation information. > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19100) Fix Spotbugs warnings in the build
[ https://issues.apache.org/jira/browse/HADOOP-19100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823054#comment-17823054 ] Shilun Fan commented on HADOOP-19100: - [~ayushtkn] Thanks for initiating this JIRA! I will follow up by fixing some spotbugs in the yarn module. > Fix Spotbugs warnings in the build > -- > > Key: HADOOP-19100 > URL: https://issues.apache.org/jira/browse/HADOOP-19100 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ayush Saxena >Priority: Major > > We are getting spotbugs warnings in every PR. > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-common-project-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html] > [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-root-warnings.html] > > Source: > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/console -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19099) Add Protobuf Compatibility Notes
[ https://issues.apache.org/jira/browse/HADOOP-19099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-19099. - Fix Version/s: 3.4.0 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add Protobuf Compatibility Notes > > > Key: HADOOP-19099 > URL: https://issues.apache.org/jira/browse/HADOOP-19099 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > > In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version > 3.21.12. This version may have compatibility issues with certain versions of > JDK8. We will document this situation in the index.md file of hadoop-3.4.0 > and inform users that we will discontinue support for JDK8 in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19099) Add Protobuf Compatibility Notes
Shilun Fan created HADOOP-19099: --- Summary: Add Protobuf Compatibility Notes Key: HADOOP-19099 URL: https://issues.apache.org/jira/browse/HADOOP-19099 Project: Hadoop Common Issue Type: Sub-task Components: documentation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version 3.21.12. This version may have compatibility issues with certain versions of JDK8. We will document this situation in the index.md file of hadoop-3.4.0 and inform users that we will discontinue support for JDK8 in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17302) Upgrade to jQuery 3.5.1 in hadoop-sls
[ https://issues.apache.org/jira/browse/HADOOP-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17302: Component/s: build common Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Upgrade to jQuery 3.5.1 in hadoop-sls > - > > Key: HADOOP-17302 > URL: https://issues.apache.org/jira/browse/HADOOP-17302 > Project: Hadoop Common > Issue Type: Improvement > Components: build, common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Aryan Gupta >Assignee: Aryan Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > jQuery is upgraded from 3.3.1 to 3.5.1 at > hadoop/hadoop-tools/hadoop-sls/src/main/html/js/thirdparty -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19084) prune dependency exports of hadoop-* modules
[
https://issues.apache.org/jira/browse/HADOOP-19084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved HADOOP-19084.
-
Hadoop Flags: Reviewed
Resolution: Fixed
> prune dependency exports of hadoop-* modules
>
>
> Key: HADOOP-19084
> URL: https://issues.apache.org/jira/browse/HADOOP-19084
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.4.0, 3.5.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> this is probably caused by HADOOP-18613:
> ZK is pulling in some extra transitive stuff which surfaces in applications
> which import hadoop-common into their poms. It doesn't seem to show up in our
> distro, but downstream you get warnings about duplicate logging stuff
> {code}
> | +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
> | | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
> | | | \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile -
> omitted for duplicate)
> | | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
> | | +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for
> conflict with 4.1.100.Final)
> | | +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile -
> omitted for conflict with 4.1.100.Final)
> | | +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
> | | +- ch.qos.logback:logback-core:jar:1.2.10:compile
> | | +- ch.qos.logback:logback-classic:jar:1.2.10:compile
> | | | +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for
> duplicate)
> | | | \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict
> with 1.7.30)
> | | \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict
> with 2.14.0)
> {code}
> proposed: exclude the zk dependencies we either override outselves or don't
> need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19084) prune dependency exports of hadoop-* modules
[
https://issues.apache.org/jira/browse/HADOOP-19084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-19084:
Fix Version/s: 3.4.0
> prune dependency exports of hadoop-* modules
>
>
> Key: HADOOP-19084
> URL: https://issues.apache.org/jira/browse/HADOOP-19084
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.4.0, 3.5.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> this is probably caused by HADOOP-18613:
> ZK is pulling in some extra transitive stuff which surfaces in applications
> which import hadoop-common into their poms. It doesn't seem to show up in our
> distro, but downstream you get warnings about duplicate logging stuff
> {code}
> | +- org.apache.zookeeper:zookeeper:jar:3.8.3:compile
> | | +- org.apache.zookeeper:zookeeper-jute:jar:3.8.3:compile
> | | | \- (org.apache.yetus:audience-annotations:jar:0.12.0:compile -
> omitted for duplicate)
> | | +- org.apache.yetus:audience-annotations:jar:0.12.0:compile
> | | +- (io.netty:netty-handler:jar:4.1.94.Final:compile - omitted for
> conflict with 4.1.100.Final)
> | | +- (io.netty:netty-transport-native-epoll:jar:4.1.94.Final:compile -
> omitted for conflict with 4.1.100.Final)
> | | +- (org.slf4j:slf4j-api:jar:1.7.30:compile - omitted for duplicate)
> | | +- ch.qos.logback:logback-core:jar:1.2.10:compile
> | | +- ch.qos.logback:logback-classic:jar:1.2.10:compile
> | | | +- (ch.qos.logback:logback-core:jar:1.2.10:compile - omitted for
> duplicate)
> | | | \- (org.slf4j:slf4j-api:jar:1.7.32:compile - omitted for conflict
> with 1.7.30)
> | | \- (commons-io:commons-io:jar:2.11.0:compile - omitted for conflict
> with 2.14.0)
> {code}
> proposed: exclude the zk dependencies we either override outselves or don't
> need.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18088) Replace log4j 1.x with reload4j
[ https://issues.apache.org/jira/browse/HADOOP-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18088: Target Version/s: 3.3.3, 3.3.5, 3.2.3, 2.10.2, 3.4.0 (was: 2.10.2, 3.2.3, 3.3.5, 3.3.3) > Replace log4j 1.x with reload4j > --- > > Key: HADOOP-18088 > URL: https://issues.apache.org/jira/browse/HADOOP-18088 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3, 3.5.0, 3.4.1 > > Time Spent: 8h > Remaining Estimate: 0h > > As proposed in the dev mailing list > (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's > replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x > and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18088) Replace log4j 1.x with reload4j
[ https://issues.apache.org/jira/browse/HADOOP-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18088: Fix Version/s: 3.4.0 > Replace log4j 1.x with reload4j > --- > > Key: HADOOP-18088 > URL: https://issues.apache.org/jira/browse/HADOOP-18088 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3, 3.5.0, 3.4.1 > > Time Spent: 8h > Remaining Estimate: 0h > > As proposed in the dev mailing list > (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's > replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x > and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822729#comment-17822729 ] Shilun Fan commented on HADOOP-19090: - [~ste...@apache.org] Thank you for your contribution and support! > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822321#comment-17822321 ] Shilun Fan edited comment on HADOOP-19090 at 3/1/24 1:12 AM: - [~pj.fanning] [~ste...@apache.org] I am not opposed to releasing hadoop-thirdparty-1.2.1, but I don't think now is a good time to do so. If we were to release hadoop-thirdparty-1.2.1, our process is too lengthy: 1. We need to announce this in a public mailing list. 2. Then initiate a vote, and after the vote passes, release hadoop-thirdparty-1.2.1. 3. Introduce version 1.2.1 in the Hadoop trunk branch. 4. backport hadoop-3.4.0 Even if we upgrade to protobuf-3.23.4, there might still be other issues. If there really are other issues, would we need to release hadoop-thirdparty-1.2.2? I think a better approach would be: To notify about this in the release email for hadoop-3.4.0, and then release hadoop-thirdparty-1.2.1 before the release of hadoop-3.4.1, followed by thorough validation. cc: [~hexiaoqiao] was (Author: slfan1989): [~pj.fanning] [~ste...@apache.org] I am not opposed to releasing hadoop-thirdparty-1.2.1, but I don't think now is a good time to do so. If we were to release hadoop-thirdparty-1.2.1, our process is too lengthy: 1. We need to announce this in a public mailing list. 2. Then initiate a vote, and after the vote passes, release hadoop-thirdparty-1.2.1. 3. Introduce version 1.2.1 in the Hadoop trunk branch. 4. backport hadoop-3.4.0 Even if we upgrade to protobuf-3.23.4, there might still be other issues. If there really are other issues, would we need to release hadoop-thirdparty-1.2.2? I think a better approach would be: To notify about this in the release email for hadoop-3.4.0, and then release hadoop-thirdparty-1.2.1 before the release of hadoop-3.4.1, followed by thorough validation. > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822321#comment-17822321 ] Shilun Fan commented on HADOOP-19090: - [~pj.fanning] [~ste...@apache.org] I am not opposed to releasing hadoop-thirdparty-1.2.1, but I don't think now is a good time to do so. If we were to release hadoop-thirdparty-1.2.1, our process is too lengthy: 1. We need to announce this in a public mailing list. 2. Then initiate a vote, and after the vote passes, release hadoop-thirdparty-1.2.1. 3. Introduce version 1.2.1 in the Hadoop trunk branch. 4. backport hadoop-3.4.0 Even if we upgrade to protobuf-3.23.4, there might still be other issues. If there really are other issues, would we need to release hadoop-thirdparty-1.2.2? I think a better approach would be: To notify about this in the release email for hadoop-3.4.0, and then release hadoop-thirdparty-1.2.1 before the release of hadoop-3.4.1, followed by thorough validation. > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822318#comment-17822318 ] Shilun Fan commented on HADOOP-19090: - > ok, do we need to cut a new release of thirdparty module for the next RC? If > so, I can kick that off? Can we upgrade hadoop-thirdparty-1.2.1 before hadoop-3.4.1? This requires public mail discussion. Can we release hadoop-3.4.0 first? > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822315#comment-17822315 ] Shilun Fan commented on HADOOP-19090: - [~pj.fanning] Steve suggested an alternative solution in HADOOP-19018, where we can ship as is with a warning note. cc: [~hexiaoqiao] > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822315#comment-17822315 ] Shilun Fan edited comment on HADOOP-19090 at 2/29/24 10:14 PM: --- [~pj.fanning] Steve suggested an alternative solution in HADOOP-19018, where we can ship as is with a warning note. cc: [~hexiaoqiao] was (Author: slfan1989): [~pj.fanning] Steve suggested an alternative solution in HADOOP-19018, where we can ship as is with a warning note. cc: [~hexiaoqiao] > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822105#comment-17822105
]
Shilun Fan commented on HADOOP-19018:
-
Gentle ping. [~ste...@apache.org]
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821932#comment-17821932
]
Shilun Fan edited comment on HADOOP-19018 at 2/29/24 5:28 AM:
--
[~ste...@apache.org] after careful consideration of Steve's question, can we
address it in hadoop-3.4.1 and release hadoop-thirdparty-1.2.1 before
hadoop-3.4.1?
I am preparing hadoop-3.4.0-RC3, which will include two backported PRs from
branch-3.4 compared to hadoop-3.4.0-RC2:
1. HADOOP-18088: Replacing log4j 1.x with reload4j.
2. HADOOP-19084: Pruning hadoop-common transitive dependencies.
Additionally, I will use hadoop-release-support to package the ARM version.
I would like to hear the thoughts of other members. If there are no further
issues, I will proceed with packaging hadoop-3.4.0-RC3 and initiate a community
vote. I hope that after hadoop-3.4.0-RC3, we can release hadoop-3.4.0.
cc: [~hexiaoqiao]
was (Author: slfan1989):
[~ste...@apache.org] after careful consideration of Steve's question, can we
address it in hadoop-3.4.1 and release hadoop-thirdparty-1.2.1 before
hadoop-3.4.1?
I am preparing hadoop-3.4.0-RC3, which will include two backported PRs from
branch-3.4 compared to xx-3.4.0-RC2:
1. HADOOP-18088: Replacing log4j 1.x with reload4j.
2. HADOOP-19084: Pruning hadoop-common transitive dependencies.
Additionally, I will use hadoop-release-support to package the ARM version.
I would like to hear the thoughts of other members. If there are no further
issues, I will proceed with packaging hadoop-3.4.0-RC3 and initiate a community
vote. I hope that after hadoop-3.4.0-RC3, we can release hadoop-3.4.0.
cc: [~hexiaoqiao]
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821932#comment-17821932
]
Shilun Fan commented on HADOOP-19018:
-
[~ste...@apache.org] after careful consideration of Steve's question, can we
address it in hadoop-3.4.1 and release hadoop-thirdparty-1.2.1 before
hadoop-3.4.1?
I am preparing hadoop-3.4.0-RC3, which will include two backported PRs from
branch-3.4 compared to xx-3.4.0-RC2:
1. HADOOP-18088: Replacing log4j 1.x with reload4j.
2. HADOOP-19084: Pruning hadoop-common transitive dependencies.
Additionally, I will use hadoop-release-support to package the ARM version.
I would like to hear the thoughts of other members. If there are no further
issues, I will proceed with packaging hadoop-3.4.0-RC3 and initiate a community
vote. I hope that after hadoop-3.4.0-RC3, we can release hadoop-3.4.0.
cc: [~hexiaoqiao]
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19018) Release Hadoop 3.4.0
[
https://issues.apache.org/jira/browse/HADOOP-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821634#comment-17821634
]
Shilun Fan commented on HADOOP-19018:
-
[~ste...@apache.org] Thank you for addressing this issue! I can assist in
releasing thirdparty-1.2.1 to resolve it. cc [~hexiaoqiao]
> Release Hadoop 3.4.0
>
>
> Key: HADOOP-19018
> URL: https://issues.apache.org/jira/browse/HADOOP-19018
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>
> Confirmed features to be included in the release:
> - Enhanced functionality for YARN Federation.
> - Redesigned resource allocation in YARN Capacity Scheduler
> - Optimization of HDFS RBF.
> - Introduction of fine-grained global locks for DataNodes.
> - Improvements in the stability of HDFS EC, and more.
> - Fixes for important CVEs.
> *Issues that need to be addressed in hadoop-3.4.0-RC0 version.*
> 1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the
> version setting is correct.
> 2. confirm the highlight of hadoop-3.4.0.
> 3. backport branch-3.4.0/branch-3.4.
> {code:java}
> HADOOP-19040. mvn site commands fails due to MetricsSystem And
> MetricsSystemImpl changes.
> YARN-11634. [Addendum] Speed-up TestTimelineClient.
> MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
>
> Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19065) Update Protocol Buffers installation to 3.21.12
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19065: Fix Version/s: 3.4.1 > Update Protocol Buffers installation to 3.21.12 > > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Zhaobo Huang >Assignee: Zhaobo Huang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > Update docs and docker script to cover downloading the 3.21.12 protobuf > compiler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19065) Update Protocol Buffers installation to 3.21.12
[ https://issues.apache.org/jira/browse/HADOOP-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19065: Fix Version/s: 3.4.0 > Update Protocol Buffers installation to 3.21.12 > > > Key: HADOOP-19065 > URL: https://issues.apache.org/jira/browse/HADOOP-19065 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Zhaobo Huang >Assignee: Zhaobo Huang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0 > > > Update docs and docker script to cover downloading the 3.21.12 protobuf > compiler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819667#comment-17819667 ] Shilun Fan edited comment on HADOOP-15984 at 2/22/24 2:24 PM: -- Here's some slightly better news: I believe I've mostly resolved the issue with Guice and Jersey2 conflicts causing injection failures. I've opted for a different approach to resolve it, and so far, it seems viable. I'll submit the PR as soon as possible. cc: [~aajisaka] [~ste...@apache.org] [~vjasani] [~fanningpj] was (Author: slfan1989): Here's some slightly better news: I believe I've mostly resolved the issue with Guice and Jersey2 conflicts causing injection failures. I've opted for a different approach to resolve it, and so far, it seems viable. I'll submit the PR as soon as possible. cc: [~aajisaka][~ste...@apache.org][~vjasani][~fanningpj] > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819667#comment-17819667 ] Shilun Fan commented on HADOOP-15984: - Here's some slightly better news: I believe I've mostly resolved the issue with Guice and Jersey2 conflicts causing injection failures. I've opted for a different approach to resolve it, and so far, it seems viable. I'll submit the PR as soon as possible. cc: [~aajisaka][~ste...@apache.org][~vjasani][~fanningpj] > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19071) Update maven-surefire-plugin from 3.0.0 to 3.2.5
[ https://issues.apache.org/jira/browse/HADOOP-19071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818749#comment-17818749 ] Shilun Fan commented on HADOOP-19071: - [~ayushtkn] Thank you for bringing this issue to our attention. I will carefully compare the test reports and provide a response accordingly. cc: [~ste...@apache.org] > Update maven-surefire-plugin from 3.0.0 to 3.2.5 > - > > Key: HADOOP-19071 > URL: https://issues.apache.org/jira/browse/HADOOP-19071 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, common >Affects Versions: 3.4.0, 3.5.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19076) move jersey code in hadoop-common jar to a new hadoop-jersey1-common jar
[ https://issues.apache.org/jira/browse/HADOOP-19076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817693#comment-17817693 ] Shilun Fan commented on HADOOP-19076: - Thank you very much for your thoughts! it should be difficult for us to directly upgrade to jersey3, it is documented in HADOOP-15984. Jersey3 is not compatible w/Jetty9, then we should upgrade Jetty11 as well, which means dropping support of Java8. https://dailydevsblog.com/troubleshoot/resolved-how-to-configure-jersey-3x-to-work-with-embedded-jetty-9x-141995/ > move jersey code in hadoop-common jar to a new hadoop-jersey1-common jar > > > Key: HADOOP-19076 > URL: https://issues.apache.org/jira/browse/HADOOP-19076 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Priority: Major > > Hadoop's Jersey dependencies are causing us real trouble. > I'm wondering if it would be a good idea to take the jersey and javax.ws code > out of hadoop-common and move it into a dedicated hadoop-jersey1-common jar. > We could later create a hadoop-jersey2-common (or hadoop-jersey3-common - > because Jersey 3 is out and maybe better to skip to Jersey 2). > hadoop-jersey1-common and hadoop-jersey2-common would have equivalent classes > - just depend on different versions of Jersey. > Example code: > * > https://github.com/apache/hadoop/blob/12498b35bbb754225b0b2ca90d5ad4f5cf628d56/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java#L1030 > * > https://github.com/apache/hadoop/blob/12498b35bbb754225b0b2ca90d5ad4f5cf628d56/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java#L89 > Hadoop modules that need access to the common jersey code could start with > depending on hadoop-jersey1-common but later be refactored to use > hadoop-jersey2-common. We could do this on a module by module basis (one a > time). > hadoop-common jar would have its jersey and jsr311-api dependencies removed. > Wdyt [~slfan1989], [~steve_l], [~ayushsaxena] ? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817023#comment-17817023
]
Shilun Fan edited comment on HADOOP-15984 at 2/13/24 3:01 PM:
--
[~iwasakims] [~fanningpj] Thank you for the comments! I encountered some issues
while debugging locally because jsr and rs-api define some classes with similar
names but not identical content, resulting in conflicts. Indeed, Jersey1 and
Jersey2 cannot coexist.
{code:java}
// jersey1 need jsr311
javax.ws.rs
jsr311-api
1.1.1
// jersey2 need rs-api 2.x
javax.ws.rs
javax.ws.rs-api
2.1.1
{code}
class loading resolution is one way to solve it, but it's quite challenging. We
can opt for a simpler solution, such as hadoop-thirdparty-jersey2, using shade
to rename the packages of the dependency jars. This approach is much simpler.
The benefit of using thirdparty-jersey2 is that we can attempt to upgrade
Jersey on a particular module without affecting other modules.
cc: [~ste...@apache.org]
was (Author: slfan1989):
[~iwasakims] [~fanningpj] Thank you for the comments! I encountered some issues
while debugging locally because jsr and rs-api define some classes with similar
names but not identical content, resulting in conflicts. Indeed, Jersey1 and
Jersey2 cannot coexist.
{code:java}
// jersey1 need jsr311
javax.ws.rs
jsr311-api
1.1.1
// jersey2 need rs-api 2.x
javax.ws.rs
javax.ws.rs-api
2.1.1
{code}
class loading resolution is one way to solve it, but it's quite challenging. We
can opt for a simpler solution, such as hadoop-thirdparty-jersey2, using shade
to rename the packages of the dependency jars. This approach is much simpler.
cc: [~ste...@apache.org]
> Update jersey from 1.19 to 2.x
> --
>
> Key: HADOOP-15984
> URL: https://issues.apache.org/jira/browse/HADOOP-15984
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817023#comment-17817023
]
Shilun Fan commented on HADOOP-15984:
-
[~iwasakims] [~fanningpj] Thank you for the comments! I encountered some issues
while debugging locally because jsr and rs-api define some classes with similar
names but not identical content, resulting in conflicts. Indeed, Jersey1 and
Jersey2 cannot coexist.
{code:java}
// jersey1 need jsr311
javax.ws.rs
jsr311-api
1.1.1
// jersey2 need rs-api 2.x
javax.ws.rs
javax.ws.rs-api
2.1.1
{code}
class loading resolution is one way to solve it, but it's quite challenging. We
can opt for a simpler solution, such as hadoop-thirdparty-jersey2, using shade
to rename the packages of the dependency jars. This approach is much simpler.
cc: [~ste...@apache.org]
> Update jersey from 1.19 to 2.x
> --
>
> Key: HADOOP-15984
> URL: https://issues.apache.org/jira/browse/HADOOP-15984
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19075) add shaded jersey jar hadoop-shaded-jersey
Shilun Fan created HADOOP-19075: --- Summary: add shaded jersey jar hadoop-shaded-jersey Key: HADOOP-19075 URL: https://issues.apache.org/jira/browse/HADOOP-19075 Project: Hadoop Common Issue Type: Improvement Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan We need to address the issue of upgrading Jersey, replacing Jersey 1.x with Jersey 2.x. Jersey 2 and Jersey 1 are completely different, with many differences, and the same project cannot simultaneously introduce Jersey 2 and Jersey 1. In order to allow the project to reference both Jersey 2 and Jersey 1 simultaneously, facilitating the step-by-step module upgrade process, I plan to introduce Jersey 2 into hadoop-thirdparty. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[ https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816651#comment-17816651 ] Shilun Fan commented on HADOOP-15984: - Thank you all for continuously following up on this JIRA, especially [~aajisaka] and [~vjasani] who have made significant contributions. Recently, I have been reading the comments in the JIRA and examining related PRs. The current issue is that Jersey 1 and Jersey 2 are completely different components, and we need to fundamentally refactor all the content of the YARN web service. This is a massive undertaking, but we still need to continue pushing forward and completing it. The reason this JIRA is particularly challenging is that the Hadoop project cannot simultaneously support Jersey 1 and Jersey 2. I am considering introducing Jersey 2 into Hadoop third-party dependencies and shading the related paths to ensure that Jersey 1 and Jersey 2 can be introduced into the Hadoop project simultaneously. Then, we can proceed with modifying each module one by one. I would like to hear everyone's thoughts on this. [~aajisaka] [~vjasani] [~ayushtkn] > Update jersey from 1.19 to 2.x > -- > > Key: HADOOP-15984 > URL: https://issues.apache.org/jira/browse/HADOOP-15984 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > jersey-json 1.19 depends on Jackson 1.9.2. Let's upgrade. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19070) Java 17 compile support
[ https://issues.apache.org/jira/browse/HADOOP-19070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816647#comment-17816647 ] Shilun Fan commented on HADOOP-19070: - I agree with Steve's idea, and I will start following up on HADOOP-15984. Thank you all for your attention to this JIRA. I hope we can successfully complete the Java 17 support work. > Java 17 compile support > --- > > Key: HADOOP-19070 > URL: https://issues.apache.org/jira/browse/HADOOP-19070 > Project: Hadoop Common > Issue Type: Task >Reporter: Shilun Fan >Priority: Major > > We plan to make Hadoop compatible with Java 17. There is a lot of work to be > done, but we still aim to complete this task by 2024. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18980) S3A credential provider remapping: make extensible
[ https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816645#comment-17816645 ] Shilun Fan commented on HADOOP-18980: - [~vjasani] Thank you for the information! I have checked and confirmed that your submitted PR is included in hadoop-3.4.0-RC2. The vote for hadoop-3.4.0-RC2 has begun. If you have any questions, please feel free to ask at any time. > S3A credential provider remapping: make extensible > -- > > Key: HADOOP-18980 > URL: https://issues.apache.org/jira/browse/HADOOP-18980 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > s3afs will now remap the common com.amazonaws credential providers to > equivalents in the v2 sdk or in hadoop-aws > We could do the same for third party credential providers by taking a > key=value list in a configuration property and adding to the map. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[ https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18975: Fix Version/s: 3.4.0 > AWS SDK v2: extend support for FIPS endpoints > -- > > Key: HADOOP-18975 > URL: https://issues.apache.org/jira/browse/HADOOP-18975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > v1 SDK supported FIPS just by changing the endpoint. > Now we have a new builder setting to use. > * add new fs.s3a.endpoint.fips option > * pass it down > * test -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19004) S3A: Support Authentication through HttpSigner API
[
https://issues.apache.org/jira/browse/HADOOP-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-19004:
Fix Version/s: 3.4.0
> S3A: Support Authentication through HttpSigner API
> ---
>
> Key: HADOOP-19004
> URL: https://issues.apache.org/jira/browse/HADOOP-19004
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> The latest AWS SDK changes how signing works, and for signing S3Express
> signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is
> needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-19027:
Fix Version/s: 3.4.0
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0, 3.4.1
>
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19015) Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool
[ https://issues.apache.org/jira/browse/HADOOP-19015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19015: Fix Version/s: 3.4.0 > Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting > for connection from pool > -- > > Key: HADOOP-19015 > URL: https://issues.apache.org/jira/browse/HADOOP-19015 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.7, 3.5.0, 3.4.1 > > > Getting errors in jobs which can be fixed by increasing this > 2023-12-14 17:35:56,602 [ERROR] [TezChild] |tez.TezProcessor|: > java.lang.RuntimeException: java.io.IOException: > org.apache.hadoop.net.ConnectTimeoutException: getFileStatus on > s3a://aaa/cc-hive-jzv5y6/warehouse/tablespace/managed/hive/student/delete_delta_012_012_0001/bucket_1_0: > software.amazon.awssdk.core.exception.SdkClientException: Unable to execute > HTTP request: Timeout waiting for connection from pool at > org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206) > at > org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) > at > org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:84) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:70) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:70) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:40) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptible -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19033) S3A: disable checksum validation
[ https://issues.apache.org/jira/browse/HADOOP-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-19033: Fix Version/s: 3.4.0 > S3A: disable checksum validation > > > Key: HADOOP-19033 > URL: https://issues.apache.org/jira/browse/HADOOP-19033 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.5.0, 3.4.1 > > > AWS v2 sdk turns on client-side checksum validation; this kills performance > Given we are using TLS to download from AWS s3, there's implicit channel > checksumming going on on, that's along with the IPv4 TCP checksumming. > We don't need it, all it does is slow us down. > proposed: disable in DefaultS3ClientFactory > I don't want to add an option to enable it as it only complicates life (yet > another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18125) Utility to identify git commit / Jira fixVersion discrepancies for RC preparation
[ https://issues.apache.org/jira/browse/HADOOP-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18125: Component/s: build Hadoop Flags: Reviewed Target Version/s: 3.3.3, 3.4.0 (was: 3.3.3) Affects Version/s: 3.3.3 3.2.3 2.10.2 3.4.0 > Utility to identify git commit / Jira fixVersion discrepancies for RC > preparation > - > > Key: HADOOP-18125 > URL: https://issues.apache.org/jira/browse/HADOOP-18125 > Project: Hadoop Common > Issue Type: Task > Components: build >Affects Versions: 3.4.0, 2.10.2, 3.2.3, 3.3.3 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > As part of RC preparation, we need to identify all git commits that landed > on release branch, however their corresponding Jira is either not resolved > yet or does not contain expected fixVersions. Only when we have git commits > and corresponding Jiras with expected fixVersion resolved, we get all such > Jiras included in auto-generated CHANGES.md as per Yetus changelog generator. > Proposal of this Jira is to provide such script that can be useful for all > upcoming RC preparations and list down all Jiras where we need manual > intervention. This utility script should use Jira API to retrieve individual > fields and use git log to loop through commit history. > The script should identify these issues: > # commit is reverted as per commit message > # commit does not contain Jira number format (e.g. HADOOP- / HDFS- > etc) in message > # Jira does not have expected fixVersion > # Jira has expected fixVersion, but it is not yet resolved > # Jira has release corresponding fixVersion and is resolved, but no > corresponding commit yet found > It can take inputs as: > # First commit hash to start excluding commits from history > # Fix Version > # JIRA Project Name > # Path of project's working dir > # Jira server url -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18014) CallerContext should not include some characters
[ https://issues.apache.org/jira/browse/HADOOP-18014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18014: Component/s: ipc Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.4.0 Affects Version/s: 3.3.5 3.4.0 > CallerContext should not include some characters > > > Key: HADOOP-18014 > URL: https://issues.apache.org/jira/browse/HADOOP-18014 > Project: Hadoop Common > Issue Type: Task > Components: ipc >Affects Versions: 3.4.0, 3.3.5 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18061) Update the year to 2022
[ https://issues.apache.org/jira/browse/HADOOP-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18061: Component/s: common Target Version/s: 3.3.2, 3.2.3, 2.10.2, 3.4.0 Affects Version/s: 3.3.2 3.2.3 2.10.2 3.4.0 > Update the year to 2022 > --- > > Key: HADOOP-18061 > URL: https://issues.apache.org/jira/browse/HADOOP-18061 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0, 2.10.2, 3.2.3, 3.3.2 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 40m > Remaining Estimate: 0h > > Update the year to 2022 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17946) Update commons-lang to 3.12.0
[ https://issues.apache.org/jira/browse/HADOOP-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17946: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > Update commons-lang to 3.12.0 > - > > Key: HADOOP-17946 > URL: https://issues.apache.org/jira/browse/HADOOP-17946 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.3.2 >Reporter: Sean Busbey >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > our commons-lang3 dependency is currently 3.7, which is nearly 4 years old. > latest right now is 3.12 and there are at least some fixes that would make us > more robust on JDKs newer than openjdk8 (e.g. LANG-1384. [release notes > indicate 3.9 is the first to support > jdk11|https://commons.apache.org/proper/commons-lang/changes-report.html]). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17955) Bump netty to the latest 4.1.68
[ https://issues.apache.org/jira/browse/HADOOP-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17955: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Bump netty to the latest 4.1.68 > --- > > Key: HADOOP-17955 > URL: https://issues.apache.org/jira/browse/HADOOP-17955 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Netty 4.1.68 fixes the following vulnerabilities. > * Bzip2Decoder doesn't allow setting size restrictions for decompressed data > (#CVE-2021-37136) > * SnappyFrameDecoder doesn't restrict chunk length any may buffer skippable > chunks in an unnecessary way (#CVE-2021-37137) > For more details: [https://netty.io/news/2021/09/09/4-1-68-Final.html] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17834) Bump aliyun-sdk-oss to 3.13.0
[ https://issues.apache.org/jira/browse/HADOOP-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17834: Component/s: build common Target Version/s: 3.3.2, 3.2.3, 3.4.0 Affects Version/s: 3.3.2 3.2.3 3.4.0 > Bump aliyun-sdk-oss to 3.13.0 > - > > Key: HADOOP-17834 > URL: https://issues.apache.org/jira/browse/HADOOP-17834 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0, 3.2.3, 3.3.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Bump aliyun-sdk-oss to 3.13.0 in order to remove transitive dependency on > jdom 1.1. > Ref: > https://issues.apache.org/jira/browse/HADOOP-17820?focusedCommentId=17390206=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17390206. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17743) Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects
[ https://issues.apache.org/jira/browse/HADOOP-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17743: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, > hadoop-tools and cloud-storage projects > - > > Key: HADOOP-17743 > URL: https://issues.apache.org/jira/browse/HADOOP-17743 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17683) Update commons-io to 2.8.0
[ https://issues.apache.org/jira/browse/HADOOP-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17683: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Update commons-io to 2.8.0 > -- > > Key: HADOOP-17683 > URL: https://issues.apache.org/jira/browse/HADOOP-17683 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > https://nvd.nist.gov/vuln/detail/CVE-2021-29425 > In Apache Commons IO before 2.7, When invoking the method > FileNameUtils.normalize with an improper input string, like "//../foo", or > "\\..\foo", the result would be the same value, thus possibly providing > access to files in the parent directory, but not further above (thus > "limited" path traversal), if the calling code would use the result to > construct a path value. > We don't use this API in the Hadoop code, but it's still good to update > anyway (we're on 2.5, which is 4 years old) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17625) Update to Jetty 9.4.39
[ https://issues.apache.org/jira/browse/HADOOP-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17625: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Update to Jetty 9.4.39 > -- > > Key: HADOOP-17625 > URL: https://issues.apache.org/jira/browse/HADOOP-17625 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17624) Remove any rocksdb exclusion code
[ https://issues.apache.org/jira/browse/HADOOP-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17624: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 > Remove any rocksdb exclusion code > - > > Key: HADOOP-17624 > URL: https://issues.apache.org/jira/browse/HADOOP-17624 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > RocksDB was added as part of Ozone runtime, and we added code to exclude > RocksDB from the core runtime. Now that Ozone is a separate project, we > should remove anything referencing RocksDB now. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17622) Avoid usage of deprecated IOUtils#cleanup API
[ https://issues.apache.org/jira/browse/HADOOP-17622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17622: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Avoid usage of deprecated IOUtils#cleanup API > - > > Key: HADOOP-17622 > URL: https://issues.apache.org/jira/browse/HADOOP-17622 > Project: Hadoop Common > Issue Type: Task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We can replace usage of deprecated API IOUtils#cleanup() with > IOUtils#cleanupWithLogger(). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17614) Bump netty to the latest 4.1.61
[ https://issues.apache.org/jira/browse/HADOOP-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17614: Component/s: build common Target Version/s: 3.2.3, 3.3.1, 3.4.0 > Bump netty to the latest 4.1.61 > --- > > Key: HADOOP-17614 > URL: https://issues.apache.org/jira/browse/HADOOP-17614 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.3.1, 3.4.0, 3.2.3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2.5h > Remaining Estimate: 0h > > For more details: https://netty.io/news/2021/03/09/4-1-60-Final.html > Actually, just yesterday there's a new version 4.1.61. > https://netty.io/news/2021/03/30/4-1-61-Final.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17030) Remove unused joda-time
[ https://issues.apache.org/jira/browse/HADOOP-17030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17030: Component/s: build common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Remove unused joda-time > --- > > Key: HADOOP-17030 > URL: https://issues.apache.org/jira/browse/HADOOP-17030 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.4.0 > > Attachments: HADOOP-17030.001.patch > > > Joda-time is defined in the hadoop-project/pom.xml but it's not used > anywhere. It should be easy to remove it without problems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16990) Update Mockserver
[ https://issues.apache.org/jira/browse/HADOOP-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16990: Component/s: hdfs-client Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.2.2, 3.4.0 Affects Version/s: 3.3.1 3.2.2 3.4.0 > Update Mockserver > - > > Key: HADOOP-16990 > URL: https://issues.apache.org/jira/browse/HADOOP-16990 > Project: Hadoop Common > Issue Type: Task > Components: hdfs-client >Affects Versions: 3.2.2, 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Attila Doroszlai >Priority: Major > Fix For: 3.2.2, 3.3.1, 3.4.0 > > Attachments: HADOOP-16990-branch-3.1.004.patch, > HADOOP-16990-branch-3.3.002.patch, HADOOP-16990.001.patch, > HDFS-15620-branch-3.3-addendum.patch > > > We are on Mockserver 3.9.2 which is more than 5 years old. Time to update. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16866) Upgrade spotbugs to 4.0.6
[ https://issues.apache.org/jira/browse/HADOOP-16866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16866: Component/s: build command Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Upgrade spotbugs to 4.0.6 > - > > Key: HADOOP-16866 > URL: https://issues.apache.org/jira/browse/HADOOP-16866 > Project: Hadoop Common > Issue Type: Task > Components: build, command >Affects Versions: 3.4.0 >Reporter: Tsuyoshi Ozawa >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.4.0 > > > [https://github.com/spotbugs/spotbugs/releases] > spotbugs 4.0.0 is now released. > > We can upgrade spotbugs' version to: > 1. 3.1.12 (conservative option) > 2. 4.0.0 (which might includes incompatible changes, according to the > migration guide: [https://spotbugs.readthedocs.io/en/stable/migration.html]) > > Step by step approach is also acceptable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17234) Add .asf.yaml to allow github and jira integration
[ https://issues.apache.org/jira/browse/HADOOP-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17234: Component/s: build common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Add .asf.yaml to allow github and jira integration > -- > > Key: HADOOP-17234 > URL: https://issues.apache.org/jira/browse/HADOOP-17234 > Project: Hadoop Common > Issue Type: Task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > As of now the default for github is set only to worklog, To enable link and > label, We need to add this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17055) Remove residual code of Ozone
[ https://issues.apache.org/jira/browse/HADOOP-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17055: Component/s: ozone common Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Remove residual code of Ozone > - > > Key: HADOOP-17055 > URL: https://issues.apache.org/jira/browse/HADOOP-17055 > Project: Hadoop Common > Issue Type: Task > Components: common, ozone >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18005) Correct log format for LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18005: Component/s: security Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > Correct log format for LdapGroupsMapping > > > Key: HADOOP-18005 > URL: https://issues.apache.org/jira/browse/HADOOP-18005 > Project: Hadoop Common > Issue Type: Wish > Components: security >Affects Versions: 3.4.0, 3.3.2 >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Attachments: image-2021-11-12-09-20-58-968.png, > image-2021-11-12-09-22-12-454.png > > Time Spent: 50m > Remaining Estimate: 0h > > Correct log format for LdapGroupsMapping. > !image-2021-11-12-09-22-12-454.png|width=557,height=101! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15348) S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly
[
https://issues.apache.org/jira/browse/HADOOP-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-15348:
---
Assignee: Steve Loughran
> S3A Input Stream bytes read counter isn't getting through to
> StorageStatistics/instrumentation properly
> ---
>
> Key: HADOOP-15348
> URL: https://issues.apache.org/jira/browse/HADOOP-15348
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.4.0
>
>
> TL;DR: we should have common storage statistics for bytes read and bytes
> written, and S3A should use them in its instrumentation and have enum names
> to match.
> # in the S3AInputStream we call
> {{S3AInstrumentation.StreamStatistics.bytesRead(long)}}, which adds the
> amount to {{bytesRead}}, in a read(), readFully, or forward seek() reading in
> data
> # and in {{S3AInstrumentation.mergeInputStreamStatistics}}, that is pulled
> into streamBytesRead.
> # which has a Statistics name of ""stream_bytes_read"
> # but that is served up in the Storage statistics as
> "STREAM_SEEK_BYTES_READ", which is the wrong name.
> # and there isn't a common name for the counter across other filesystems.
> For now: people can use the wrong name in the enum; we may want to think
> about retaining it when adding the correct name. And maybe add a
> @Evolving/@LimitedPrivate scope pair to the enum
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18159) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
[
https://issues.apache.org/jira/browse/HADOOP-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-18159:
---
Assignee: André F.
> Certificate doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> --
>
> Key: HADOOP-18159
> URL: https://issues.apache.org/jira/browse/HADOOP-18159
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.1, 3.3.2, 3.3.3
> Environment: hadoop 3.3.1
> httpclient 4.5.13
> JDK8
>Reporter: André F.
>Assignee: André F.
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. If you see this error message when trying to use s3a:// or gs:// URLs,
> look for copies of cos_api-bundle.jar on your classpath and remove them.
> Libraries which include shaded apache httpclient libraries
> (hadoop-client-runtime.jar, aws-java-sdk-bundle.jar,
> gcs-connector-shaded.jar, cos_api-bundle.jar) all load and use the unshaded
> resource mozilla/public-suffix-list.txt. If an out of date version of this is
> found on the classpath first, attempts to negotiate TLS connections may fail
> with the error "Certificate doesn't match any of the subject alternative
> names".
> In a hadoop installation, you can use the findclass tool to track down where
> the public-suffix-list.txt is coming from.
> {code}
> hadoop org.apache.hadoop.util.FindClass locate mozilla/public-suffix-list.txt
> {code}
> So far, the cos_api-bundle-5.6.19.jar appears to be the source of this
> problem.
>
> h2. bug report
> Trying to run any job after bumping our Spark version (which is now using
> Hadoop 3.3.1), lead us to the current exception while reading files on s3:
> {code:java}
> org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
> s3a:///.parquet: com.amazonaws.SdkClientException: Unable to
> execute HTTP request: Certificate for doesn't match
> any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]:
> Unable to execute HTTP request: Certificate for doesn't match any of
> the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208) at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3351)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4277)
> at {code}
>
> {code:java}
> Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> doesn't match any of the subject alternative names:
> [*.s3.amazonaws.com, s3.amazonaws.com]
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
> at
> com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> at
> com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
> at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
> at com.amazonaws.http.conn.$Proxy16.connect(Unknown Source)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at
> com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at
> com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
>
[jira] [Assigned] (HADOOP-17142) Fix outdated properties of journal node when perform rollback
[
https://issues.apache.org/jira/browse/HADOOP-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-17142:
---
Assignee: Deegue
> Fix outdated properties of journal node when perform rollback
> -
>
> Key: HADOOP-17142
> URL: https://issues.apache.org/jira/browse/HADOOP-17142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Deegue
>Assignee: Deegue
>Priority: Minor
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
>
> When rollback HDFS cluster, properties in JNStorage won't be refreshed after
> the storage dir changed. It leads to exceptions when starting namenode.
> The exception like:
> {code:java}
> 2020-07-09 19:04:12,810 FATAL [IPC Server handler 105 on 8022]
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error:
> recoverUnfinalizedSegments failed for required journal
> (JournalAndStream(mgr=QJM to [10.0.118.217:8485, 10.0.117.208:8485,
> 10.0.118.179:8485], stream=null))
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
> exceptions to achieve quorum size 2/3. 3 exceptions thrown:
> 10.0.118.217:8485: Incompatible namespaceID for journal Storage Directory
> /mnt/vdc-11176G-0/dfs/jn/nameservicetest1: NameNode has nsId 647617129 but
> storage has nsId 0
> at
> org.apache.hadoop.hdfs.qjournal.server.JNStorage.checkConsistentNamespace(JNStorage.java:236)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.newEpoch(Journal.java:300)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.newEpoch(JournalNodeRpcServer.java:136)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.newEpoch(QJournalProtocolServerSideTranslatorPB.java:133)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25417)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2274)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18118) Fix KMS Accept Queue Size default value to 500
[ https://issues.apache.org/jira/browse/HADOOP-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18118: --- Assignee: guophilipse > Fix KMS Accept Queue Size default value to 500 > -- > > Key: HADOOP-18118 > URL: https://issues.apache.org/jira/browse/HADOOP-18118 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > From HADOOP-15638,`hadoop.http.socket.backlog.size` was set as 500 by default > ,we can change code default value to keep consistent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16930) Add com.amazonaws.auth.profile.ProfileCredentialsProvider to hadoop-aws docs
[
https://issues.apache.org/jira/browse/HADOOP-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan reassigned HADOOP-16930:
---
Assignee: Nicholas Chammas
> Add com.amazonaws.auth.profile.ProfileCredentialsProvider to hadoop-aws docs
>
>
> Key: HADOOP-16930
> URL: https://issues.apache.org/jira/browse/HADOOP-16930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs/s3
>Affects Versions: 3.2.1
>Reporter: Nicholas Chammas
>Assignee: Nicholas Chammas
>Priority: Minor
> Fix For: 3.4.0
>
>
> There is a very, very useful S3A authentication method that is not currently
> documented: {{com.amazonaws.auth.profile.ProfileCredentialsProvider}}
> This provider lets you source your AWS credentials from a shared credentials
> file, typically stored under {{~/.aws/credentials}}, using a [named
> profile|https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html].
> All you need is to set the {{AWS_PROFILE}} environment variable, and the
> provider will get the appropriate credentials for you.
> I discovered this from my coworkers, but cannot find it in the docs for
> hadoop-aws. I'd expect to see it at least mentioned in [this
> section|https://hadoop.apache.org/docs/r2.9.2/hadoop-aws/tools/hadoop-aws/index.html#S3A_Authentication_methods].
> It should probably be added to the docs for every minor release that
> supports it, which I'd guess includes 2.8 on up.
> (This provider should probably also be added to the default list of
> credential provider classes, but we can address that in another ticket. I can
> say that at least in 2.9.2, it's not in the default list.)
> (This is not to be confused with
> {{com.amazonaws.auth.InstanceProfileCredentialsProvider}}, which serves a
> completely different purpose.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-15327: Component/s: common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Upgrade MR ShuffleHandler to use Netty4 > --- > > Key: HADOOP-15327 > URL: https://issues.apache.org/jira/browse/HADOOP-15327 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Xiaoyu Yao >Assignee: Szilard Nemeth >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HADOOP-15327.001.patch, HADOOP-15327.002.patch, > HADOOP-15327.003.patch, HADOOP-15327.004.patch, HADOOP-15327.005.patch, > HADOOP-15327.005.patch, > getMapOutputInfo_BlockingOperationException_awaitUninterruptibly.log, > hades-results-20221108.zip, testfailure-testMapFileAccess-emptyresponse.zip, > testfailure-testReduceFromPartialMem.zip > > Time Spent: 11.5h > Remaining Estimate: 0h > > This way, we can remove the dependencies on the netty3 (jboss.netty) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17526) Use Slf4jRequestLog for HttpRequestLog
[ https://issues.apache.org/jira/browse/HADOOP-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17526: Component/s: common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Use Slf4jRequestLog for HttpRequestLog > -- > > Key: HADOOP-17526 > URL: https://issues.apache.org/jira/browse/HADOOP-17526 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Akira Ajisaka >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Now the log file name and retain days in HttpRequestLog is configured from > log4j.properties. It can be configured from Configuration instead to drop the > usage of log4j API. > Update > Use jetty's Slf4jRequestLogWriter, we can remove the code dependency on log > appenders. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18018) unguava: remove Preconditions from hadoop-tools modules
[ https://issues.apache.org/jira/browse/HADOOP-18018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18018: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > unguava: remove Preconditions from hadoop-tools modules > --- > > Key: HADOOP-18018 > URL: https://issues.apache.org/jira/browse/HADOOP-18018 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Replace guava Preconditions by internal implementations that rely on java8+ > APIs in the hadoop.util for all modules in hadoop-tools. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17970) unguava: remove Preconditions from hdfs-projects module
[ https://issues.apache.org/jira/browse/HADOOP-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17970: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > unguava: remove Preconditions from hdfs-projects module > --- > > Key: HADOOP-17970 > URL: https://issues.apache.org/jira/browse/HADOOP-17970 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Replace guava Preconditions by internal implementations that rely on java8+ > APIs in the hadoop.util for all modules in hadoop-hdfs-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17952) Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-common-project modules
[ https://issues.apache.org/jira/browse/HADOOP-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17952: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.2.3, 2.10.2, 3.3.1, 3.4.0 Affects Version/s: 3.2.3 2.10.2 3.3.1 3.4.0 > Replace Guava VisibleForTesting by Hadoop's own annotation in > hadoop-common-project modules > --- > > Key: HADOOP-17952 > URL: https://issues.apache.org/jira/browse/HADOOP-17952 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0, 2.10.2, 3.2.3 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 2.10.2, 3.2.3 > > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17951) AccessPoint verifyBucketExistsV2 always returns false
[ https://issues.apache.org/jira/browse/HADOOP-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17951: Component/s: fs/s3 > AccessPoint verifyBucketExistsV2 always returns false > - > > Key: HADOOP-17951 > URL: https://issues.apache.org/jira/browse/HADOOP-17951 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Bogdan Stolojan >Assignee: Bogdan Stolojan >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Turns out the implementation added for accesspoints to check that they exist > was always returning "false". (Is it an improvement from the SDK which always > returned "true"? nope). > The fix for this is trivial as we need to check for the presence of a message > "Could not access through this access point" or 404 if the AP doesn't exist. > Otherwise it does. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17930) implement non-guava Precondition checkState
[
https://issues.apache.org/jira/browse/HADOOP-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17930:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.2.4, 3.3.2, 3.4.0
> implement non-guava Precondition checkState
> ---
>
> Key: HADOOP-17930
> URL: https://issues.apache.org/jira/browse/HADOOP-17930
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Affects Versions: 3.4.0, 3.2.3, 3.3.2
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to add the implementation {{checkState}} to the existing class
> {{org.apache.hadoop.util.Preconditions}}
> +The plan is as follows+
> * implement {{org.apache.hadoop.util.Preconditions.checkState}} with the
> minimum set of interface used in the current hadoop repo.
> * we can replace {{guava.Preconditions}} by
> {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been
> implemented (both this jira and HADOOP-17929 are complete).
> * We need the change to be easily to be backported in 3.x.
> previous jiras:
> * HADOOP-17126 was created to implement CheckNotNull.
> * HADOOP-17929 implementing checkArgument.
> CC: [~ste...@apache.org], [~vjasani]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17951) AccessPoint verifyBucketExistsV2 always returns false
[ https://issues.apache.org/jira/browse/HADOOP-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17951: Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > AccessPoint verifyBucketExistsV2 always returns false > - > > Key: HADOOP-17951 > URL: https://issues.apache.org/jira/browse/HADOOP-17951 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Bogdan Stolojan >Assignee: Bogdan Stolojan >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Turns out the implementation added for accesspoints to check that they exist > was always returning "false". (Is it an improvement from the SDK which always > returned "true"? nope). > The fix for this is trivial as we need to check for the presence of a message > "Could not access through this access point" or 404 if the AP doesn't exist. > Otherwise it does. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17929) implement non-guava Precondition checkArgument
[
https://issues.apache.org/jira/browse/HADOOP-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17929:
Component/s: command
Hadoop Flags: Reviewed
Target Version/s: 3.2.4, 3.3.2, 3.4.0
> implement non-guava Precondition checkArgument
> --
>
> Key: HADOOP-17929
> URL: https://issues.apache.org/jira/browse/HADOOP-17929
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: command
>Affects Versions: 3.4.0, 3.2.3, 3.3.2
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As part In order to replace Guava Preconditions, we need to implement our own
> versions of the API.
> This Jira is to add the implementation {{checkArgument}} to the existing
> class {{org.apache.hadoop.util.Preconditions}}
> +The plan is as follows+
> * implement {{org.apache.hadoop.util.Preconditions.checkArgument}} with the
> minimum set of interface used in the current hadoop repo.
> * we can replace {{guava.Preconditions}} by
> {{org.apache.hadoop.util.Preconditions}} once all the interfaces have been
> implemented.
> * We need the change to be easily to be backported in 3.x.
> A previous jira HADOOP-17126 was created to replace CheckNotNull.
> HADOOP-17930 is created to implement checkState.
> CC: [~ste...@apache.org], [~vjasani]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17910) [JDK 17] TestNetUtils fails
[
https://issues.apache.org/jira/browse/HADOOP-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17910:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> [JDK 17] TestNetUtils fails
> ---
>
> Key: HADOOP-17910
> URL: https://issues.apache.org/jira/browse/HADOOP-17910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> TestNetUtils#testInvalidAddress fails.
> {noformat}
> [INFO] Running org.apache.hadoop.net.TestNetUtils
> [ERROR] Tests run: 48, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 4.469 s <<< FAILURE! - in org.apache.hadoop.net.TestNetUtils
> [ERROR] testInvalidAddress(org.apache.hadoop.net.TestNetUtils) Time elapsed:
> 0.386 s <<< FAILURE!
> java.lang.AssertionError:
> Expected to find 'invalid-test-host:0' but got unexpected exception:
> java.net.UnknownHostException: invalid-test-host/:0
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:592)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:551)
> at
> org.apache.hadoop.net.TestNetUtils.testInvalidAddress(TestNetUtils.java:109)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> at
> org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:396)
> at
> org.apache.hadoop.test.GenericTestUtils.assertExceptionContains(GenericTestUtils.java:373)
> at
> org.apache.hadoop.net.TestNetUtils.testInvalidAddress(TestNetUtils.java:116)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[jira] [Updated] (HADOOP-17814) Provide fallbacks for identity/cost providers and backoff enable
[ https://issues.apache.org/jira/browse/HADOOP-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17814: Component/s: ipc Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Provide fallbacks for identity/cost providers and backoff enable > > > Key: HADOOP-17814 > URL: https://issues.apache.org/jira/browse/HADOOP-17814 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > This sub-task is to provide default properties for identity-provider.impl, > cost-provider.impl and backoff.enable such that if properties with port is > not configured, we can fallback to default property (port-less). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16272) Update HikariCP to 4.0.3
[ https://issues.apache.org/jira/browse/HADOOP-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-16272: Component/s: build common Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Update HikariCP to 4.0.3 > > > Key: HADOOP-16272 > URL: https://issues.apache.org/jira/browse/HADOOP-16272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, common >Affects Versions: 3.4.0 >Reporter: Yuming Wang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17715) ABFS: Append blob tests with non HNS accounts fail
[ https://issues.apache.org/jira/browse/HADOOP-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17715: Component/s: fs/azure Hadoop Flags: Reviewed Target Version/s: 3.3.2, 3.4.0 Affects Version/s: 3.3.2 3.4.0 > ABFS: Append blob tests with non HNS accounts fail > -- > > Key: HADOOP-17715 > URL: https://issues.apache.org/jira/browse/HADOOP-17715 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0, 3.3.2 >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Append blob tests with non HNS accounts fail. > # The script to run the tests should ensure that append blob tests with non > HNS account don't execute > # Should have proper documentation mentioning that append blob is allowed > only for HNS accounts -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17331) [JDK 15] TestDNS fails by UncheckedIOException
[
https://issues.apache.org/jira/browse/HADOOP-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-17331:
Component/s: test
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> [JDK 15] TestDNS fails by UncheckedIOException
> --
>
> Key: HADOOP-17331
> URL: https://issues.apache.org/jira/browse/HADOOP-17331
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: test
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After [JDK-8235783|https://bugs.openjdk.java.net/browse/JDK-8235783],
> DatagramSocket::connect throws UncheckedIOException if connect fails.
> {noformat}
> [INFO] Running org.apache.hadoop.net.TestDNS
> [ERROR] Tests run: 12, Failures: 0, Errors: 5, Skipped: 0, Time elapsed:
> 0.403 s <<< FAILURE! - in org.apache.hadoop.net.TestDNS
> [ERROR] testNullDnsServer(org.apache.hadoop.net.TestDNS) Time elapsed: 0.134
> s <<< ERROR!
> java.io.UncheckedIOException: java.net.SocketException: Unsupported address
> type
> at
> java.base/sun.nio.ch.DatagramSocketAdaptor.connect(DatagramSocketAdaptor.java:120)
> at java.base/java.net.DatagramSocket.connect(DatagramSocket.java:341)
> {noformat}
> Full error log:
> https://gist.github.com/aajisaka/2a24cb2b110cc3d19f7dec6256db6844
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17666) Update LICENSE for 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17666: Component/s: common > Update LICENSE for 3.3.1 > > > Key: HADOOP-17666 > URL: https://issues.apache.org/jira/browse/HADOOP-17666 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available, release-blocker > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Before release, do another round of check for the LICENSE file to make sure > the dependency versions are updated correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17666) Update LICENSE for 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17666: Affects Version/s: 3.3.1 3.4.0 > Update LICENSE for 3.3.1 > > > Key: HADOOP-17666 > URL: https://issues.apache.org/jira/browse/HADOOP-17666 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1, 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: pull-request-available, release-blocker > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Before release, do another round of check for the LICENSE file to make sure > the dependency versions are updated correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17665) Ignore missing keystore configuration in reloading mechanism
[ https://issues.apache.org/jira/browse/HADOOP-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-17665: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.3.1, 3.4.0 Affects Version/s: 3.3.1 3.4.0 > Ignore missing keystore configuration in reloading mechanism > - > > Key: HADOOP-17665 > URL: https://issues.apache.org/jira/browse/HADOOP-17665 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.3.1, 3.4.0 >Reporter: Borislav Iordanov >Assignee: Borislav Iordanov >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When there is no configuration of keystore/truststore location, the reload > mechanism should be disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
