[jira] [Resolved] (HADOOP-13147) Constructors must not call overrideable methods in PureJavaCrc32C
[ https://issues.apache.org/jira/browse/HADOOP-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-13147. --- Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Constructors must not call overrideable methods in PureJavaCrc32C > - > > Key: HADOOP-13147 > URL: https://issues.apache.org/jira/browse/HADOOP-13147 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.6-alpha > Environment: > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32C.java >Reporter: Sebb >Assignee: Sebb >Priority: Blocker > Labels: pull-request-available > Fix For: 3.5.0 > > > Constructors must not call overrideable methods. > An object is not guaranteed fully constructed until the constructor exits, so > the subclass override may not see the fully created parent object. > This applies to: > PureJavaCrc32 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19166) [DOC] Drop Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x
Ayush Saxena created HADOOP-19166: - Summary: [DOC] Drop Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x Key: HADOOP-19166 URL: https://issues.apache.org/jira/browse/HADOOP-19166 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Reading the docs, found this page, which is pretty irrelevant in current context or upcoming 3.x releases, can explore dropping it https://apache.github.io/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19165) Explore dropping protobuf 2.5.0 from the distro
Ayush Saxena created HADOOP-19165: - Summary: Explore dropping protobuf 2.5.0 from the distro Key: HADOOP-19165 URL: https://issues.apache.org/jira/browse/HADOOP-19165 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena explore if protobuf-2.5.0 can be dropped from distro, it is a transitive dependency from HBase, but HBase doesn't use it in the code. Check if it is the only one pulling it into the distro & if things break we exclude that, if none let get rid of it -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19164) Hadoop CLI MiniCluster is broken
Ayush Saxena created HADOOP-19164:
-
Summary: Hadoop CLI MiniCluster is broken
Key: HADOOP-19164
URL: https://issues.apache.org/jira/browse/HADOOP-19164
Project: Hadoop Common
Issue Type: Bug
Reporter: Ayush Saxena
Documentation is also broken & it doesn't work either
(https://apache.github.io/hadoop/hadoop-project-dist/hadoop-common/CLIMiniCluster.html)
*Fails with:*
{noformat}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/mockito/stubbing/Answer
at
org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2666)
at
org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2680)
at
org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:588)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 9 more{noformat}
{*}Command executed:{*}
{noformat}
bin/mapred minicluster -format{noformat}
*Documentation Issues:*
{noformat}
bin/mapred minicluster -rmport RM_PORT -jhsport JHS_PORT{noformat}
Without -format option it doesn't work the first time telling Namenode isn't
formatted, So, this should be corrected.
{noformat}
2024-05-04 00:35:52,933 WARN namenode.FSNamesystem: Encountered exception
loading fsimage
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:253)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19107) Drop support for HBase v1
[ https://issues.apache.org/jira/browse/HADOOP-19107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19107. --- Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Drop support for HBase v1 > - > > Key: HADOOP-19107 > URL: https://issues.apache.org/jira/browse/HADOOP-19107 > Project: Hadoop Common > Issue Type: Task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Drop support for Hbase V1 and make building Hbase v2 default. > Dev List: > [https://lists.apache.org/thread/vb2gh5ljwncbrmqnk0oflb8ftdz64hhs] > https://lists.apache.org/thread/o88hnm7q8n3b4bng81q14vsj3fbhfx5w -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19130) FTPFileSystem rename with full qualified path broken
[
https://issues.apache.org/jira/browse/HADOOP-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-19130.
---
Fix Version/s: 3.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> FTPFileSystem rename with full qualified path broken
>
>
> Key: HADOOP-19130
> URL: https://issues.apache.org/jira/browse/HADOOP-19130
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 0.20.2, 3.3.3, 3.3.4, 3.3.6
>Reporter: shawn
>Assignee: shawn
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
> Attachments: image-2024-03-27-09-59-12-381.png,
> image-2024-03-28-09-58-19-721.png
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When use fs shell to put/rename file in ftp server with full qualified
> path , it always get "Input/output error"(eg.
> [ftp://user:password@localhost/pathxxx]), the reason is that
> changeWorkingDirectory command underneath is being passed a string with
> [file://|file:///] uri prefix which will not be understand by ftp server
> !image-2024-03-27-09-59-12-381.png|width=948,height=156!
>
> in our case, after
> client.changeWorkingDirectory("ftp://mytest:myt...@10.5.xx.xx/files;)
> executed, the workingDirectory of ftp server is still "/", which is
> incorrect(not understand by ftp server)
> !image-2024-03-28-09-58-19-721.png|width=745,height=431!
> the solution should be pass
> absoluteSrc.getParent().toUri().getPath().toString to avoid
> [file://|file:///] uri prefix, like this:
> {code:java}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }{code}
> already related issue as follows
> https://issues.apache.org/jira/browse/HADOOP-8653
> I create this issue and add related unit test.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19123) Update commons-configuration2 to 2.10.1 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-19123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19123. --- Fix Version/s: 3.5.0 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Update commons-configuration2 to 2.10.1 due to CVE > -- > > Key: HADOOP-19123 > URL: https://issues.apache.org/jira/browse/HADOOP-19123 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0, 3.4.1 > > > https://github.com/advisories/GHSA-9w38-p64v-xpmv -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19077) Remove use of javax.ws.rs.core.HttpHeaders
[ https://issues.apache.org/jira/browse/HADOOP-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19077. --- Fix Version/s: 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Remove use of javax.ws.rs.core.HttpHeaders > -- > > Key: HADOOP-19077 > URL: https://issues.apache.org/jira/browse/HADOOP-19077 > Project: Hadoop Common > Issue Type: Task > Components: io >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.1 > > > One step towards removing Hadoop's dependence on Jersey1 and jsr311-api. > We have other classes where we can get HTTP header names. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19024) Use bouncycastle jdk18 1.77
[ https://issues.apache.org/jira/browse/HADOOP-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19024. --- Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Use bouncycastle jdk18 1.77 > --- > > Key: HADOOP-19024 > URL: https://issues.apache.org/jira/browse/HADOOP-19024 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > They have stopped patching the JDK 1.5 jars that Hadoop uses (see > https://issues.apache.org/jira/browse/HADOOP-18540). > The new artifacts have similar names - but the names are like bcprov-jdk18on > as opposed to bcprov-jdk15on. > CVE-2023-33201 is an example of a security issue that seems only to be fixed > in the JDK 1.8 artifacts (ie no JDK 1.5 jar has the fix). > https://www.bouncycastle.org/releasenotes.html#r1rv77 latest current release > but the CVE was fixed in 1.74. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19107) Drop support for HBase v1
Ayush Saxena created HADOOP-19107: - Summary: Drop support for HBase v1 Key: HADOOP-19107 URL: https://issues.apache.org/jira/browse/HADOOP-19107 Project: Hadoop Common Issue Type: Task Reporter: Ayush Saxena Drop support for Hbase V1 and make building Hbase v2 default. Dev List: [https://lists.apache.org/thread/vb2gh5ljwncbrmqnk0oflb8ftdz64hhs] https://lists.apache.org/thread/o88hnm7q8n3b4bng81q14vsj3fbhfx5w -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19090) Update Protocol Buffers installation to 3.23.4
[ https://issues.apache.org/jira/browse/HADOOP-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19090. --- Fix Version/s: 3.4.1 Hadoop Flags: Reviewed Resolution: Fixed > Update Protocol Buffers installation to 3.23.4 > -- > > Key: HADOOP-19090 > URL: https://issues.apache.org/jira/browse/HADOOP-19090 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We are seeing issues with Java 8 usage of protobuf-java > See https://issues.apache.org/jira/browse/HADOOP-18197 and comments about > java.lang.NoSuchMethodError: > java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19100) Fix Spotbugs warnings in the build
Ayush Saxena created HADOOP-19100: - Summary: Fix Spotbugs warnings in the build Key: HADOOP-19100 URL: https://issues.apache.org/jira/browse/HADOOP-19100 Project: Hadoop Common Issue Type: Bug Reporter: Ayush Saxena We are getting spotbugs warnings in every PR. [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-common-project-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-hdfs-project-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html] [https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/artifact/out/branch-spotbugs-root-warnings.html] Source: https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1517/console -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19020) Update the year to 2024
[ https://issues.apache.org/jira/browse/HADOOP-19020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-19020. --- Fix Version/s: 3.4.0 2.10.3 3.3.7 Hadoop Flags: Reviewed Resolution: Fixed > Update the year to 2024 > --- > > Key: HADOOP-19020 > URL: https://issues.apache.org/jira/browse/HADOOP-19020 > Project: Hadoop Common > Issue Type: Task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.3, 3.3.7 > > > Update the year to 2024 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19020) Update the year to 2024
Ayush Saxena created HADOOP-19020: - Summary: Update the year to 2024 Key: HADOOP-19020 URL: https://issues.apache.org/jira/browse/HADOOP-19020 Project: Hadoop Common Issue Type: Task Reporter: Ayush Saxena Assignee: Ayush Saxena Update the year to 2024 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18613) Upgrade ZooKeeper to version 3.8.3
[ https://issues.apache.org/jira/browse/HADOOP-18613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18613. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Upgrade ZooKeeper to version 3.8.3 > -- > > Key: HADOOP-18613 > URL: https://issues.apache.org/jira/browse/HADOOP-18613 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.4 >Reporter: Tamas Penzes >Assignee: Bilwa S T >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18924) Upgrade grpc jars to v1.53.0 due to CVEs
[ https://issues.apache.org/jira/browse/HADOOP-18924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18924. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Upgrade grpc jars to v1.53.0 due to CVEs > > > Key: HADOOP-18924 > URL: https://issues.apache.org/jira/browse/HADOOP-18924 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > https://mvnrepository.com/artifact/io.grpc/grpc-protobuf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18957. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18964) Update plugin for SBOM generation to 2.7.10 #6235
[ https://issues.apache.org/jira/browse/HADOOP-18964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18964. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update plugin for SBOM generation to 2.7.10 #6235 > - > > Key: HADOOP-18964 > URL: https://issues.apache.org/jira/browse/HADOOP-18964 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Vinod Anandan >Assignee: Vinod Anandan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Update the CycloneDX Maven plugin for SBOM generation to 2.7.10 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18974) Replace commons-lang imports with commons-lang3
Ayush Saxena created HADOOP-18974: - Summary: Replace commons-lang imports with commons-lang3 Key: HADOOP-18974 URL: https://issues.apache.org/jira/browse/HADOOP-18974 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena we have a direct dependency on commons-lang3 but still we are using commons-lang at a couple of places, which is a transitive dependency coming from hbase, solr, and couple of other places. good to replace those with lang3, rather than relying on older transitive dependency -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18963) Fix typos in .gitignore
[ https://issues.apache.org/jira/browse/HADOOP-18963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18963. --- Fix Version/s: 3.4.0 (was: 3.3.6) Hadoop Flags: Reviewed Resolution: Fixed > Fix typos in .gitignore > --- > > Key: HADOOP-18963 > URL: https://issues.apache.org/jira/browse/HADOOP-18963 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.6 >Reporter: 袁焊忠 >Assignee: 袁焊忠 >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > DS_Store is auto generated by Mac in every opened folder, which is useless > but annoying. Not only DS_Store file in the repository root directory should > be ignored but DS_Store file in its subfolders. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18905) Negative timeout in ZKFailovercontroller due to overflow
[ https://issues.apache.org/jira/browse/HADOOP-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18905. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Negative timeout in ZKFailovercontroller due to overflow > > > Key: HADOOP-18905 > URL: https://issues.apache.org/jira/browse/HADOOP-18905 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.6 >Reporter: ConfX >Assignee: ConfX >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Graceful fence timeout of FailoverController in ZKFailovercontroller equals > to `ha.failover-controller.graceful-fence.rpc-timeout.ms` * 2. Since users > are unaware of this calculation, it thus has risks of overflowing to a > negative number if users set > `ha.failover-controller.graceful-fence.rpc-timeout.ms` to a large value. > > To reproduce: > 1. set `ha.failover-controller.graceful-fence.rpc-timeout.ms` to 1092752431 > 2. run `mvn surefire:test > -Dtest=org.apache.hadoop.ha.TestZKFailoverController#testGracefulFailoverFailBecomingStandby` > > We create a PR that provides a fix by checking the timeout after > multiplication is at least 0. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18936) Upgrade to jetty 9.4.53
[ https://issues.apache.org/jira/browse/HADOOP-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18936. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Upgrade to jetty 9.4.53 > --- > > Key: HADOOP-18936 > URL: https://issues.apache.org/jira/browse/HADOOP-18936 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > 2 CVE fixes in > https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.53.v20231009 > 4 more security fixes in > https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.52.v20230823 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18916) module-info classes from external dependencies appearing in uber jars
[ https://issues.apache.org/jira/browse/HADOOP-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18916. --- Fix Version/s: 3.4.0 3.3.9 Hadoop Flags: Reviewed Resolution: Fixed > module-info classes from external dependencies appearing in uber jars > - > > Key: HADOOP-18916 > URL: https://issues.apache.org/jira/browse/HADOOP-18916 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > hadoop-client-minicluster and hadoop-client-runtime try unsuccessfully to > exclude module-info classes from dependencies. Over time, more and more jars > have these classes. > The module-info classes are causing issue with CI builds. Builds can fail if > there are more than module-inf class that is not excluded. > It seems better to exclude them all, especially since they will be affected > by shading anyway. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18934) Add documentation for building hadoop with docker
Ayush Saxena created HADOOP-18934: - Summary: Add documentation for building hadoop with docker Key: HADOOP-18934 URL: https://issues.apache.org/jira/browse/HADOOP-18934 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena We have documentation for Single Node setup & stuff, maybe good if we can add documentation to get started via docker: https://hub.docker.com/r/apache/hadoop -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18923) Switch to SPDX identifier for license name
[ https://issues.apache.org/jira/browse/HADOOP-18923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18923. --- Hadoop Flags: Reviewed Resolution: Fixed > Switch to SPDX identifier for license name > -- > > Key: HADOOP-18923 > URL: https://issues.apache.org/jira/browse/HADOOP-18923 > Project: Hadoop Common > Issue Type: Task >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.7 > > > [https://maven.apache.org/pom.html#Licenses] > "Using an [SPDX identifier|https://spdx.org/licenses/] as the license name is > recommended." > The Apache parent pom is already using this identifier -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18917) upgrade to commons-io 2.14.0
[ https://issues.apache.org/jira/browse/HADOOP-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18917. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade to commons-io 2.14.0 > > > Key: HADOOP-18917 > URL: https://issues.apache.org/jira/browse/HADOOP-18917 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The release contains some hardening of support in some areas -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18912) upgrade snappy-java to 1.1.10.4 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-18912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18912. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade snappy-java to 1.1.10.4 due to CVE > -- > > Key: HADOOP-18912 > URL: https://issues.apache.org/jira/browse/HADOOP-18912 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > follow up to HADOOP-18782 > https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18823) Add Labeler Github Action.
[ https://issues.apache.org/jira/browse/HADOOP-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18823. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add Labeler Github Action. > -- > > Key: HADOOP-18823 > URL: https://issues.apache.org/jira/browse/HADOOP-18823 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Add auto github lablel for ease of reviewing: > Pull Request Labeler Github Action Configuration: > https://github.com/marketplace/actions/labeler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18823) Add Labeler Github Action.
Ayush Saxena created HADOOP-18823: - Summary: Add Labeler Github Action. Key: HADOOP-18823 URL: https://issues.apache.org/jira/browse/HADOOP-18823 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.4.0 Reporter: Ayush Saxena Add auto github lablel for ease of reviewing: Pull Request Labeler Github Action Configuration: https://github.com/marketplace/actions/labeler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18794) ipc.server.handler.queue.size missing from core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18794. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > ipc.server.handler.queue.size missing from core-default.xml > --- > > Key: HADOOP-18794 > URL: https://issues.apache.org/jira/browse/HADOOP-18794 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > This config should be documented. It is necessary to help users gain a > clearer understanding of the usage of this property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18796) Compilation fails with triple-beam@1.4.0: The engine "node" is incompatible with this module. Expected version ">= 16.0.0". Got "12.22.1"
Ayush Saxena created HADOOP-18796:
-
Summary: Compilation fails with triple-beam@1.4.0: The engine
"node" is incompatible with this module. Expected version ">= 16.0.0". Got
"12.22.1"
Key: HADOOP-18796
URL: https://issues.apache.org/jira/browse/HADOOP-18796
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 3.4.0
Reporter: Ayush Saxena
{code:java}
2023-07-10T07:32:20.1069856Z [INFO] yarn install v1.22.5
2023-07-10T07:32:20.1435049Z [INFO] info No lockfile found.
2023-07-10T07:32:20.1489685Z [INFO] [1/4] Resolving packages...
2023-07-10T07:32:20.9138642Z [INFO] warning angular-route@1.6.10: For the
actively supported Angular, see https://www.npmjs.com/package/@angular/core.
AngularJS support has officially ended. For extended AngularJS support options,
see https://goo.gle/angularjs-path-forward.
2023-07-10T07:32:20.9433934Z [INFO] warning angular@1.6.10: For the actively
supported Angular, see https://www.npmjs.com/package/@angular/core. AngularJS
support has officially ended. For extended AngularJS support options, see
https://goo.gle/angularjs-path-forward.
2023-07-10T07:32:21.5051267Z [INFO] [2/4] Fetching packages...
2023-07-10T07:32:23.2554282Z [INFO] error triple-beam@1.4.0: The engine "node"
is incompatible with this module. Expected version ">= 16.0.0". Got "12.22.1"
2023-07-10T07:32:23.2651719Z [INFO] error Found incompatible module.
2023-07-10T07:32:23.2664307Z [INFO] info Visit
https://yarnpkg.com/en/docs/cli/install for documentation about this command.
{code}
Ref:
https://pipelines.actions.githubusercontent.com/serviceHosts/a3203f99-c5b3-4def-b81d-c5a8ebd0356d/_apis/pipelines/1/runs/294/signedlogcontent/2?urlExpires=2023-07-10T09%3A08%3A57.5185178Z=HMACV1=OHImbS%2FdsTBFKRxwRnV6jFPnumE0GxbB%2BalrVpAgmyw%3D
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18783) upgrade netty to 4.1.94 due to CVE
[ https://issues.apache.org/jira/browse/HADOOP-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18783. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade netty to 4.1.94 due to CVE > -- > > Key: HADOOP-18783 > URL: https://issues.apache.org/jira/browse/HADOOP-18783 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > https://github.com/advisories/GHSA-6mjq-h674-j845 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18713) Update solr from 8.8.2 to 8.11.2
[
https://issues.apache.org/jira/browse/HADOOP-18713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18713.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Update solr from 8.8.2 to 8.11.2
>
>
> Key: HADOOP-18713
> URL: https://issues.apache.org/jira/browse/HADOOP-18713
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Xuesen Liang
>Assignee: Xuesen Liang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Solr is used in hadoop-yarn-applications-catalog-webapp.
> {code:sh}
> $ grep solr.version . -r
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-project/pom.xml: 8.8.2
> $
> {code}
>
> Solr-8.8.2's log4j2 version is 2.13.2.
> Solr-8.11.2's log4j2 version is 2.17.1 (
> https://issues.apache.org/jira/browse/SOLR-15871 )
> Should we update solr to 8.11.2 correspondingly?
> Then maven will not try to download log4j2-2.13.2, which is forbidden by some
> maven repository.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18718) Fix several maven build warnings
[
https://issues.apache.org/jira/browse/HADOOP-18718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18718.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix several maven build warnings
>
>
> Key: HADOOP-18718
> URL: https://issues.apache.org/jira/browse/HADOOP-18718
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> {code}
> [WARNING] 'build.plugins.plugin.version' for
> org.cyclonedx:cyclonedx-maven-plugin is missing.
> {code}
> {code}
> [WARNING] Unknown keyword additionalItems - you should define your own Meta
> Schema. If the keyword is irrelevant for validation, just use a
> NonValidationKeyword
> {code}
> {code}
> [WARNING] 'build.plugins.plugin.version' for
> org.codehaus.mojo:findbugs-maven-plugin is missing
> {code}
> {code}
> [WARNING] Parameter 'requiresOnline' is unknown for plugin
> 'exec-maven-plugin:1.3.1:exec (pre-dist)'
> {code}
> {code}
> [WARNING] Parameter 'destDir' (user property 'destDir') is deprecated: No
> reason given
> {code}
> {code}
> [WARNING] Parameter 'tasks' is deprecated: Use target instead
> [WARNING] Parameter tasks is deprecated, use target instead
> {code}
> {code}
> [WARNING] Parameter 'systemProperties' is deprecated: Use
> systemPropertyVariables instead.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18207) Introduce hadoop-logging module
[ https://issues.apache.org/jira/browse/HADOOP-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reopened HADOOP-18207: --- > Introduce hadoop-logging module > --- > > Key: HADOOP-18207 > URL: https://issues.apache.org/jira/browse/HADOOP-18207 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > There are several goals here: > # Provide the ability to change log level, get log level, etc. > # Place all the appender implementation(?) > # Hide the real logging implementation. > # Later we could remove all the log4j references in other hadoop module. > # Move as much log4j usage to the module as possible. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17518) Usage of incorrect regex range A-z
[
https://issues.apache.org/jira/browse/HADOOP-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-17518.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Usage of incorrect regex range A-z
> --
>
> Key: HADOOP-17518
> URL: https://issues.apache.org/jira/browse/HADOOP-17518
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Marcono1234
>Assignee: Nishtha Shah
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> There are two cases where the regex {{A-z}} is used. I assume that is a typo
> (and should be {{A-Z}}) because {{A-z}} matches:
> - {{A-Z}}
> - {{\[}}, {{}}, {{\]}}, {{^}}, {{_}}, {{`}}
> - {{a-z}}
> Affected:
> -
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L109
> (and
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/util/Check.java#L115)
> -
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/resourcetypes/ResourceTypesTestHelper.java#L38
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18652) Path.suffix raises NullPointerException
[
https://issues.apache.org/jira/browse/HADOOP-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18652.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Path.suffix raises NullPointerException
> ---
>
> Key: HADOOP-18652
> URL: https://issues.apache.org/jira/browse/HADOOP-18652
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Patrick Grandjean
>Assignee: Patrick Grandjean
>Priority: Minor
> Fix For: 3.4.0
>
>
> Calling the Path.suffix method on root raises a NullPointerException. Tested
> with hadoop-client-api 3.3.2
> Scenario:
> {code:java}
> import org.apache.hadoop.fs.*
> Path root = new Path("/")
> root.getParent == null // true
> root.suffix("bar") // NPE is raised
> {code}
> Stack:
> {code:none}
> 23/03/03 15:13:18 ERROR Uncaught throwable from user code:
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:104)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at org.apache.hadoop.fs.Path.suffix(Path.java:361)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18359) Update commons-cli from 1.2 to 1.5.
[ https://issues.apache.org/jira/browse/HADOOP-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18359. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update commons-cli from 1.2 to 1.5. > > > Key: HADOOP-18359 > URL: https://issues.apache.org/jira/browse/HADOOP-18359 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18715) Add debug log for getting details of tokenKindMap
[
https://issues.apache.org/jira/browse/HADOOP-18715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18715.
---
Fix Version/s: 3.4.0
3.3.9
Hadoop Flags: Reviewed
Resolution: Fixed
> Add debug log for getting details of tokenKindMap
> -
>
> Key: HADOOP-18715
> URL: https://issues.apache.org/jira/browse/HADOOP-18715
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Pralabh Kumar
>Assignee: Pralabh Kumar
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Currently there is no way to know what is store in tokenKindMap. There are
> also no debug logs . I think we should add gettokenKindMap method , so that
> caller can see the content of the tokenKindMap . It will really help to debug
> .
> {code:java}
> while (tokenIdentifiers.hasNext()) {
> try {
> TokenIdentifier id = tokenIdentifiers.next();
> *tokenKindMap.put(id.getKind(), id.getClass());*
> } catch (ServiceConfigurationError | LinkageError e) {
> // failure to load a token implementation
> // log at debug and continue.
> LOG.debug("Failed to load token identifier implementation", e);
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18724) Open file fails with NumberFormatException for S3AFileSystem
Ayush Saxena created HADOOP-18724:
-
Summary: Open file fails with NumberFormatException for
S3AFileSystem
Key: HADOOP-18724
URL: https://issues.apache.org/jira/browse/HADOOP-18724
Project: Hadoop Common
Issue Type: Bug
Reporter: Ayush Saxena
Saw the trace for Hive-Iceberg, was using the old client, doesn't happen once I
upgraded.
{noformat}
Caused by: java.lang.NumberFormatException: For input string: "5783.0"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.base/java.lang.Long.parseLong(Long.java:692)
at java.base/java.lang.Long.parseLong(Long.java:817)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1601)
at
org.apache.hadoop.fs.s3a.impl.OpenFileSupport.prepareToOpenFile(OpenFileSupport.java:262)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.openFileWithOptions(S3AFileSystem.java:5219)
at
org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4753)
at
org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:196)
at
org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101)
at
org.apache.iceberg.avro.AvroIterable.getMetadata(AvroIterable.java:66){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18660) Filesystem Spelling Mistake
[ https://issues.apache.org/jira/browse/HADOOP-18660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18660. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Filesystem Spelling Mistake > --- > > Key: HADOOP-18660 > URL: https://issues.apache.org/jira/browse/HADOOP-18660 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sebastian Baunsgaard >Assignee: Sebastian Baunsgaard >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > The fs/Filesystem reports errors always containing the spelling mistake > 'fileystem' > It is not the only place in Hadoop this is the case, but this is the easiest > to fix. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18712) upgrade to jetty 9.4.51 due to cve
[ https://issues.apache.org/jira/browse/HADOOP-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18712. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade to jetty 9.4.51 due to cve > -- > > Key: HADOOP-18712 > URL: https://issues.apache.org/jira/browse/HADOOP-18712 > Project: Hadoop Common > Issue Type: Task > Components: common >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > https://github.com/advisories/GHSA-qw69-rqj8-6qw8 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18689) Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project
[ https://issues.apache.org/jira/browse/HADOOP-18689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18689. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project > > > Key: HADOOP-18689 > URL: https://issues.apache.org/jira/browse/HADOOP-18689 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.9 >Reporter: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > PR from github depandabot > https://github.com/apache/hadoop/pull/5502 > Mentions CVE: https://nvd.nist.gov/vuln/detail/CVE-2023-1436 > Creating ticket for tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18711) upgrade nimbus jwt jar due to issues in its embedded shaded json-smart code
[ https://issues.apache.org/jira/browse/HADOOP-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18711. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > upgrade nimbus jwt jar due to issues in its embedded shaded json-smart code > --- > > Key: HADOOP-18711 > URL: https://issues.apache.org/jira/browse/HADOOP-18711 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > https://github.com/apache/hadoop/pull/5549#issuecomment-1515174820 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18710) Add RPC metrics for response time
[ https://issues.apache.org/jira/browse/HADOOP-18710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18710. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add RPC metrics for response time > - > > Key: HADOOP-18710 > URL: https://issues.apache.org/jira/browse/HADOOP-18710 > Project: Hadoop Common > Issue Type: Improvement >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18597) Simplify single node instructions for creating directories for Map Reduce
[
https://issues.apache.org/jira/browse/HADOOP-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18597.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Simplify single node instructions for creating directories for Map Reduce
> -
>
> Key: HADOOP-18597
> URL: https://issues.apache.org/jira/browse/HADOOP-18597
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.4
>Reporter: Nikita Eshkeev
>Assignee: Nikita Eshkeev
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The {{mkdir}} command supports the {{-p}} option which instructs {{hdfs}} to
> create all the parent directories if needed. The single nose setup
> instructions now ask a user to create both /user and /user/
> directories explicitly, which can be simplified to creating just the
> /user/ with the help from the -p option of mkdir
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18590) Publish SBOM artifacts
[ https://issues.apache.org/jira/browse/HADOOP-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18590. --- Fix Version/s: 3.4.0 3.3.9 Hadoop Flags: Reviewed Resolution: Fixed > Publish SBOM artifacts > -- > > Key: HADOOP-18590 > URL: https://issues.apache.org/jira/browse/HADOOP-18590 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.9 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18701) Generic Build Improvements
Ayush Saxena created HADOOP-18701:
-
Summary: Generic Build Improvements
Key: HADOOP-18701
URL: https://issues.apache.org/jira/browse/HADOOP-18701
Project: Hadoop Common
Issue Type: Wish
Reporter: Ayush Saxena
Some proposed build changes.
* Add {{surefire.failIfNoSpecifiedTests}} as false in POM, else it fails if
the test specified in -Dtest isn't there in that module, creates problem when
you plan to run multiple tests across multiple sub-projects from the root of
the project.
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0:test (default-test) on
project hadoop-build-tools: No tests matching pattern
"TestServiceInterruptHandling" were executed! (Set
-Dsurefire.failIfNoSpecifiedTests=false to ignore this error.)
{noformat}
* Disable Concurrent builds: Folks push multiple commits in 5-10 mins while
pre-commit is running already, so good to discourage this.
* Add threshold to number of builds per day, saves resources for genuine PR's
against someone pushing multiple commits. (This & the above one: Copied idea
from Hive)
* Leverage Github Actions to delegate some of the tasks to them, so a bit of
parallel execution and might save time, may be explore pushing JDK-11 related
stuff to Github Actions (We don't run tests as of now for both JDK-11 & 8,
tests are for 8 only in precommit)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18689) Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project
Ayush Saxena created HADOOP-18689: - Summary: Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project Key: HADOOP-18689 URL: https://issues.apache.org/jira/browse/HADOOP-18689 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena PR from github depandabot https://github.com/apache/hadoop/pull/5502 Mentions CVE: https://nvd.nist.gov/vuln/detail/CVE-2023-1436 Creating ticket for tracking. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18681) Upgrade hadoop3 docker scripts to use 3.3.5
[ https://issues.apache.org/jira/browse/HADOOP-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18681. --- Hadoop Flags: Reviewed Resolution: Fixed > Upgrade hadoop3 docker scripts to use 3.3.5 > --- > > Key: HADOOP-18681 > URL: https://issues.apache.org/jira/browse/HADOOP-18681 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > > Hadoop 3.3.5 is released, the docker3 still points to 3.3,1, Lets upgrade it > to 3.3.5 > It doesn't work as of now due to broken link to apache-rat-0.13 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18682) Move hadoop docker scripts to dev-support
Ayush Saxena created HADOOP-18682: - Summary: Move hadoop docker scripts to dev-support Key: HADOOP-18682 URL: https://issues.apache.org/jira/browse/HADOOP-18682 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Exploratory: Coming from https://github.com/apache/hadoop/pull/5514 We have docker scripts maintained in a different branch. We can explore them to have as part of dev-support in our main source code. They can be used for new users to try the code without the headache of building and doing crazy stuff, and can help in dev testing as well -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18681) Upgrade hadoop3 docker scripts to use 3.3.5
Ayush Saxena created HADOOP-18681: - Summary: Upgrade hadoop3 docker scripts to use 3.3.5 Key: HADOOP-18681 URL: https://issues.apache.org/jira/browse/HADOOP-18681 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena Hadoop 3.3.5 is released, the docker3 still points to 3.3,1, Lets upgrade it to 3.3.5 It doesn't work as of now due to broken link to apache-rat-0.13 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18662) ListFiles with recursive fails with FNF
[
https://issues.apache.org/jira/browse/HADOOP-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18662.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> ListFiles with recursive fails with FNF
> ---
>
> Key: HADOOP-18662
> URL: https://issues.apache.org/jira/browse/HADOOP-18662
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Problem triggers in HDFS, but the change is in Hadoop-Common, Since the
> listFiles is defined in Hadoop-Common.
> Scenario:
> ListFiles With recursive:
> * Fetches a dir say /dir, which has some /dir/s1...s10
> * Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got
> deleted by that time
> * The entire operation fails with FNF
> Hive Cleaner uses listFiles with recursive true and this impacts that
> {noformat}
> 2023-03-06 07:45:48,331 ERROR
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner:
> [Cleaner-executor-thread-12]: Caught exception when cleaning, unable to
> complete cleaning of
> id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId:
> null,initiatorId: null java.io.FileNotFoundException: File
> hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
> at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
> at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
> at
> org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
> at
> org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18662) ListFiles with recursive fails with FNF
Ayush Saxena created HADOOP-18662:
-
Summary: ListFiles with recursive fails with FNF
Key: HADOOP-18662
URL: https://issues.apache.org/jira/browse/HADOOP-18662
Project: Hadoop Common
Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena
Problem triggers in HDFS, but the change is in Hadoop-Common, Since the
listFiles is defined in Hadoop-Common.
Scenario:
ListFiles With recursive:
* Fetches a dir say /dir, which has some /dir/s1...s10
* Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got
deleted by that time
* The entire operation fails with FNF
Hive Cleaner uses listFiles with recursive true and this impacts that
{noformat}
2023-03-06 07:45:48,331 ERROR org.apache.hadoop.hive.ql.txn.compactor.Cleaner:
[Cleaner-executor-thread-12]: Caught exception when cleaning, unable to
complete cleaning of
id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId:
null,initiatorId: null java.io.FileNotFoundException: File
hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550
does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
at
org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
at
org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
at
org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
at
org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
at
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
at
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18658) snakeyaml dependency: upgrade to v2.0
[ https://issues.apache.org/jira/browse/HADOOP-18658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18658. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > snakeyaml dependency: upgrade to v2.0 > - > > Key: HADOOP-18658 > URL: https://issues.apache.org/jira/browse/HADOOP-18658 > Project: Hadoop Common > Issue Type: Task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > * [https://github.com/advisories/GHSA-mjmj-j48q-9wg2] > * I don't think this needs to go in v3.3.5 - since this CVE affects part of > snakeyaml that hadoop doesn't use -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18582) No need to clean tmp files in distcp direct mode
[
https://issues.apache.org/jira/browse/HADOOP-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18582.
---
Hadoop Flags: Reviewed
Resolution: Fixed
> No need to clean tmp files in distcp direct mode
>
>
> Key: HADOOP-18582
> URL: https://issues.apache.org/jira/browse/HADOOP-18582
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.4
>Reporter: 1kang
>Assignee: 1kang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> it not necessary to do `cleanupTempFiles` while ditcp commit job in direct
> mode, because it there is no temp files in direct mode.
> This clean operation will increase the task execution time, because it will
> get the list of files in the target path. When the number of files in the
> target path is very large, this operation will be very slow.
> *note* there are two patches which need to be cherrypicked when picking this
> up; the original patch and a followup, both with HADOOP-18582 in the title
> {code}
> 3b7b79b37ae HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251)
> e8a6b2c2c4e HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18524) Deploy Hadoop trunk version website
[ https://issues.apache.org/jira/browse/HADOOP-18524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18524. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Deploy Hadoop trunk version website > --- > > Key: HADOOP-18524 > URL: https://issues.apache.org/jira/browse/HADOOP-18524 > Project: Hadoop Common > Issue Type: Task > Components: documentation >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Have a link to the running trunk documentation in the hadoop site. > A lot of projects like ozone, iceberg have a running documentation for the > master branch version. > Good to have for Hadoop as well -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18630) Add gh-pages in asf.yaml to deploy the current trunk doc
[ https://issues.apache.org/jira/browse/HADOOP-18630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18630. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add gh-pages in asf.yaml to deploy the current trunk doc > > > Key: HADOOP-18630 > URL: https://issues.apache.org/jira/browse/HADOOP-18630 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The docs are now pushed to gh-pages. > Add the entry in asf.yaml to publish as mentioned in > https://issues.apache.org/jira/browse/INFRA-24202 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18630) Add gh-pages in asf.yaml to deploy the current trunk doc
Ayush Saxena created HADOOP-18630: - Summary: Add gh-pages in asf.yaml to deploy the current trunk doc Key: HADOOP-18630 URL: https://issues.apache.org/jira/browse/HADOOP-18630 Project: Hadoop Common Issue Type: Sub-task Reporter: Ayush Saxena Assignee: Simhadri Govindappa The docs are now pushed to gh-pages. Add the entry in asf.yaml to publish as mentioned in https://issues.apache.org/jira/browse/INFRA-24202 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18604) Add compile platform in the hadoop version output
[ https://issues.apache.org/jira/browse/HADOOP-18604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18604. --- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > Add compile platform in the hadoop version output > - > > Key: HADOOP-18604 > URL: https://issues.apache.org/jira/browse/HADOOP-18604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Hadoop releases support both x86 and Aarch64, good to have a line indicating > this in the hadoop version output. > Inspired by: HDDS-7783 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18604) Add compile platform in the hadoop version output
Ayush Saxena created HADOOP-18604: - Summary: Add compile platform in the hadoop version output Key: HADOOP-18604 URL: https://issues.apache.org/jira/browse/HADOOP-18604 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena Hadoop releases support both x86 and Aarch64, good to have a line indicating this in the hadoop version output. Inspired by: HDDS-7783 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18586) Update the year to 2023
[ https://issues.apache.org/jira/browse/HADOOP-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18586. --- Fix Version/s: 3.4.0 2.10.3 3.2.5 3.3.6 Hadoop Flags: Reviewed Resolution: Fixed > Update the year to 2023 > --- > > Key: HADOOP-18586 > URL: https://issues.apache.org/jira/browse/HADOOP-18586 > Project: Hadoop Common > Issue Type: Task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.3, 3.2.5, 3.3.6 > > > Update the year to 2023 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18586) Update the year to 2023
Ayush Saxena created HADOOP-18586: - Summary: Update the year to 2023 Key: HADOOP-18586 URL: https://issues.apache.org/jira/browse/HADOOP-18586 Project: Hadoop Common Issue Type: Task Reporter: Ayush Saxena Assignee: Ayush Saxena Update the year to 2023 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18524) Deploy Hadoop trunk version website
Ayush Saxena created HADOOP-18524: - Summary: Deploy Hadoop trunk version website Key: HADOOP-18524 URL: https://issues.apache.org/jira/browse/HADOOP-18524 Project: Hadoop Common Issue Type: Bug Reporter: Ayush Saxena Have a link to the running trunk documentation in the hadoop site. A lot of projects like ozone, iceberg have a running documentation for the master branch version. Good to have for Hadoop as well -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18512) upgrade woodstox-core to 5.4.0 for security fix
[ https://issues.apache.org/jira/browse/HADOOP-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18512. --- Fix Version/s: 3.4.0 3.3.5 Hadoop Flags: Reviewed Resolution: Fixed > upgrade woodstox-core to 5.4.0 for security fix > --- > > Key: HADOOP-18512 > URL: https://issues.apache.org/jira/browse/HADOOP-18512 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.4 >Reporter: phoebe chen >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Per [issue|https://github.com/FasterXML/woodstox/issues/157], woodstox-core > 5.3.0 has security vulnerability and need to upgrade to 5.4.0 for fix. > The Hadoop Configuration classes uses woodstox to parse the XML format > (core-site.xml, ...) but > * people don't normally put in DTDs > * the XML format is not the wire format used when applications submit jobs to > the yarn resource manager. > * when parsing untrusted XML configuration files in restricted mode (eg. > oozie workflows), DTD support is already disabled -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18493) update jackson-databind 2.12.7.1 due to CVE fixes
[ https://issues.apache.org/jira/browse/HADOOP-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18493. --- Fix Version/s: 3.4.0 3.3.5 Hadoop Flags: Reviewed Resolution: Fixed > update jackson-databind 2.12.7.1 due to CVE fixes > - > > Key: HADOOP-18493 > URL: https://issues.apache.org/jira/browse/HADOOP-18493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42003] > * [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42004] > * both fixes have been backported (the CVEs themselves need to be updated to > reflect this) > * [https://github.com/FasterXML/jackson-databind/pull/3622] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18360) Update commons-csv from 1.0 to 1.9.0.
[ https://issues.apache.org/jira/browse/HADOOP-18360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18360. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update commons-csv from 1.0 to 1.9.0. > - > > Key: HADOOP-18360 > URL: https://issues.apache.org/jira/browse/HADOOP-18360 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > commons-csv 1.0 is a very old jar, mvnrepository shows this jar as the 2014 > version, I have compiled and tested locally, I think this jar can be upgraded > to commons-csv 1.9 version. > The link to the release note is as follows: > [https://commons.apache.org/proper/commons-csv/changes-report.html] > We can see that the new version fixes some issues. > I read the code used, we use header related methods. We found that many > header-related methods have been upgraded. > *Release 1.1 – 2014-11-16* > CSVFormat#withHeader doesn't work well with #printComment, add > withHeaderComments(String...). > CSVFormat.EXCEL should ignore empty header names. > *Release 1.2 – 2015-08-24* > CSVFormat.with* methods clear the header comments. > *Release 1.3 – 2016-05-09* > Add shortcut method for using first record as header to CSVFormat. > Add withHeader(Class) to CSVFormat. > CSVPrinter doesn't skip creation of header record if skipHeaderRecord is set > to true. > Add IgnoreCase option for accessing header names. > *Release 1.5 – 2017-09-03* > Fix incorrect method name 'withFirstRowAsHeader' in user guide. > *Release 1.7 – 2019-06-01* > Cannot get headers in column order from CSVRecord. > *Release 1.8 – 2020-02-01* > CSVFormat#validate() does not account for allowDuplicateHeaderNames. > A single empty header is allowed when not allowing empty column headers. > *Release 1.9.0 – 2020-07-24* > Add possibility to use ResultSet header meta data as CSV header. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16674) TestDNS.testRDNS can fail with ServiceUnavailableException
[ https://issues.apache.org/jira/browse/HADOOP-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-16674. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > TestDNS.testRDNS can fail with ServiceUnavailableException > -- > > Key: HADOOP-16674 > URL: https://issues.apache.org/jira/browse/HADOOP-16674 > Project: Hadoop Common > Issue Type: Bug > Components: common, net >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Ashutosh Gupta >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > TestDNS.testRDNS can fail in some network configurations; it is already setup > to catch and swallow these. > However it can also fail with a ServiceUnavailableException -which is not > caught. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18388) Allow dynamic groupSearchFilter in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18388.
---
Hadoop Flags: Reviewed
Resolution: Fixed
> Allow dynamic groupSearchFilter in LdapGroupsMapping
>
>
> Key: HADOOP-18388
> URL: https://issues.apache.org/jira/browse/HADOOP-18388
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: Reviewed, pull-request-available
> Fix For: 3.4.0
>
> Attachments: dynamic-filter-idea.patch
>
>
> As of now the lookupGroup() method doesn't allow to have placeholders in
> groupSearchFilter, so that can not be dynamically adjusted.
> If we have placeholders for groupSearchFilter like:
> (&(|(XYZ=\{0})(ABC=\{1}))(objectClass=posixGroup))
> This fails here:
>
> {code:java}
> groupResults =
> c.search(groupbaseDN,
> "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))",
> new Object[]{userDn},
> SEARCH_CONTROLS); {code}
> With
>
>
> {noformat}
> javax.naming.directory.InvalidSearchFilterException: number exceeds argument
> list: 1; remaining name {noformat}
>
> >>Dropped off or changed the details above which I thought won't be safe to
> >>disclose.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18441) Remove org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer
[ https://issues.apache.org/jira/browse/HADOOP-18441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18441. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Remove > org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer > > > Key: HADOOP-18441 > URL: https://issues.apache.org/jira/browse/HADOOP-18441 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > It's in hadoop-maven-plugins. > maven-shade-plugin has ServicesResourceTransformer now - seems better to use > that -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18417) Upgrade Maven Surefire plugin to 3.0.0-M7
[ https://issues.apache.org/jira/browse/HADOOP-18417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reopened HADOOP-18417: --- > Upgrade Maven Surefire plugin to 3.0.0-M7 > - > > Key: HADOOP-18417 > URL: https://issues.apache.org/jira/browse/HADOOP-18417 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Blocker > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > The Maven Surefire plugin 3.0.0-M1 doesn't always include the launcher as > part of it's setup, which can cause problems with Yarn tests. Some of the > Yarn modules use Jupiter, which may be a complicating factor. Switching to > 3.0.0-M7 fixes the issue. > This is currently blocking MAPREDUCE-7386 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18361) Update commons-net from 3.6 to 3.8.0.
[ https://issues.apache.org/jira/browse/HADOOP-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18361. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update commons-net from 3.6 to 3.8.0. > -- > > Key: HADOOP-18361 > URL: https://issues.apache.org/jira/browse/HADOOP-18361 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Current version 3.6 is almost ~5 years old > Upgrading to new release to keep up for new features and bug fixes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18404) Fix broken link to wiki help page in org.apache.hadoop.util.Shell
[ https://issues.apache.org/jira/browse/HADOOP-18404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18404. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix broken link to wiki help page in org.apache.hadoop.util.Shell > - > > Key: HADOOP-18404 > URL: https://issues.apache.org/jira/browse/HADOOP-18404 > Project: Hadoop Common > Issue Type: Bug >Reporter: Paul King >Assignee: Paul King >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When the old wiki was phased out, the link in org.apache.hadoop.util.Shell > wasn't updated. Fixing the link avoids the "Page Not Found" error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18387) Fix incorrect placeholder in hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18387. --- Hadoop Flags: Reviewed Resolution: Fixed > Fix incorrect placeholder in hadoop-common > -- > > Key: HADOOP-18387 > URL: https://issues.apache.org/jira/browse/HADOOP-18387 > Project: Hadoop Common > Issue Type: Bug >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18388) Allow dynamic groupSearchFilter in LdapGroupsMapping
Ayush Saxena created HADOOP-18388:
-
Summary: Allow dynamic groupSearchFilter in LdapGroupsMapping
Key: HADOOP-18388
URL: https://issues.apache.org/jira/browse/HADOOP-18388
Project: Hadoop Common
Issue Type: Improvement
Components: security
Reporter: Ayush Saxena
As of now the lookupGroup() method doesn't allow to have placeholders in
groupSearchFilter, so that can not be dynamically adjusted.
If we have placeholders for groupSearchFilter like:
(&(|(XYZ=\{0})(ABC=\{1}))(objectClass=posixGroup))
This fails here:
{code:java}
groupResults =
c.search(groupbaseDN,
"(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))",
new Object[]{userDn},
SEARCH_CONTROLS); {code}
With
{noformat}
javax.naming.directory.InvalidSearchFilterException: number exceeds argument
list: 1; remaining name {noformat}
** Dropped off or changed the details above which I thought won't be safe to
disclose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18358) Update commons-math3 from 3.1.1 to 3.6.1.
[ https://issues.apache.org/jira/browse/HADOOP-18358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18358. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update commons-math3 from 3.1.1 to 3.6.1. > - > > Key: HADOOP-18358 > URL: https://issues.apache.org/jira/browse/HADOOP-18358 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I found that commons-math3 can be upgraded from 3.1.1 to 3.6.1. Try to > upgrade, local compilation and verification are correct. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18381) Fix/Improve DistCp doc
Ayush Saxena created HADOOP-18381: - Summary: Fix/Improve DistCp doc Key: HADOOP-18381 URL: https://issues.apache.org/jira/browse/HADOOP-18381 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Ayush Saxena Fix/Improve stuff in DistCp doc: * Missing Headings from the index like (DistCp and Object Stores) * Add about -delete along with -update in example for Object Store usage. * Check the usage description for -update. It says it matches blocksize as well, which isn't true unless you specify -pb and if you do so, for cloud storages it would overwrite always.(AFAIK) need to double check. And if possible other improvements -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18354) Upgrade reload4j due to XXE vulnerability
[ https://issues.apache.org/jira/browse/HADOOP-18354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18354. --- Fix Version/s: 3.3.9 Hadoop Flags: Reviewed Resolution: Fixed > Upgrade reload4j due to XXE vulnerability > - > > Key: HADOOP-18354 > URL: https://issues.apache.org/jira/browse/HADOOP-18354 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > Time Spent: 50m > Remaining Estimate: 0h > > https://github.com/qos-ch/reload4j/issues/53 fixed in reload4j 1.2.22 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18335) Improve Pre-Commit Time
Ayush Saxena created HADOOP-18335: - Summary: Improve Pre-Commit Time Key: HADOOP-18335 URL: https://issues.apache.org/jira/browse/HADOOP-18335 Project: Hadoop Common Issue Type: Task Reporter: Ayush Saxena As of now the complete build time has reached ~24 hours, which makes tracking Jira with root level changes very tough. Even at module level, it is on the higher side, kind of 6 hours or more, which is typically equivalent to 1 working day hours: Some areas to explore: * Enable Parallel-Test profile for the modules which don't have it. * Explore improvements in the area of our Test setup, increase memory, number of threads, or some parallel executions? * Remove the modules or atleast disable the test suites of modules which are no longer being used or maintained (Separate Mail Chain is there for this) * Spin Two Pre-Commit jobs rather than 1 and split some task between them, kind of one for Java-11 build & Javadoc, Checkstyle and related stuff. * Figure out & remove irrelevant or similar tests. * Improve existing tests, Like reusing MiniDfsClusters or so, rather than a bunch of test spinning one for themselves. * If possible, if there are multiple modules to be tested, run those modules test in parallel (Exploratory: may lead to OOM) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18282) Add .asf.yaml to hadoop-thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18282. --- Fix Version/s: thirdparty-1.2.0 Hadoop Flags: Reviewed Resolution: Fixed > Add .asf.yaml to hadoop-thirdparty > -- > > Key: HADOOP-18282 > URL: https://issues.apache.org/jira/browse/HADOOP-18282 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > > no yaml file in thirdparty, it is dropping mails to common-dev for everything. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18282) Add .asf.yaml to hadoop-thirdparty
Ayush Saxena created HADOOP-18282: - Summary: Add .asf.yaml to hadoop-thirdparty Key: HADOOP-18282 URL: https://issues.apache.org/jira/browse/HADOOP-18282 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena no yaml file in thirdparty, it is dropping mails to common-dev for everything. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18244) Fix Hadoop-Common JavaDoc Error on branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-18244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18244. --- Fix Version/s: 3.3.4 Hadoop Flags: Reviewed Target Version/s: 3.3.4 (was: 3.4.0) Resolution: Fixed > Fix Hadoop-Common JavaDoc Error on branch-3.3 > - > > Key: HADOOP-18244 > URL: https://issues.apache.org/jira/browse/HADOOP-18244 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.0, 3.3.1, 3.3.2, 3.3.3 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.3.4 > > Time Spent: 2h > Remaining Estimate: 0h > > Fix Hadoop-Common JavaDoc Error on branch-3.3. > In the PR([#4267|https://github.com/apache/hadoop/pull/4267]) of > HADOOP-18224. Upgrade maven compiler plugin to 3.10.1, I found that > hadoop-common has a lot of javadoc compilation errors. I fixed it on the > trunk, hoping to backport these changes to branch 3.3. These changes will > ensure that hadoop common moudle javadoc will pass in JDK11 compilation. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18241) Move to Java 11
Ayush Saxena created HADOOP-18241: - Summary: Move to Java 11 Key: HADOOP-18241 URL: https://issues.apache.org/jira/browse/HADOOP-18241 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.4.0 Reporter: Ayush Saxena Assignee: Ayush Saxena https://lists.apache.org/thread/h5lmpqo2tz7tc02j44qxpwcnjzpxo0k2 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18166) Jenkins jobs intermittently failing in the end
Ayush Saxena created HADOOP-18166:
-
Summary: Jenkins jobs intermittently failing in the end
Key: HADOOP-18166
URL: https://issues.apache.org/jira/browse/HADOOP-18166
Project: Hadoop Common
Issue Type: Bug
Reporter: Ayush Saxena
The PR results are not completed sometime:
{noformat}
13:52:57 Recording test results
13:53:18 Remote call on hadoop10 failed
[Pipeline] echo
13:53:18 junit processing: java.io.IOException: Remote call on hadoop10 failed
{noformat}
Ref:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/8/consoleFull
Daily Builds also face similar issues:
{noformat}
java.io.IOException: Pipe closed after 0 cycles
at
org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126)
at
org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105)
at
hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:93)
at
hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
at
hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:104)
at
hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Caused: java.io.IOException: Backing channel 'hadoop19' is disconnected.
{noformat}
Ref:
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/810/console
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/807/console
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18082) Add debug log when RPC#Reader gets a Call
[ https://issues.apache.org/jira/browse/HADOOP-18082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reopened HADOOP-18082: --- > Add debug log when RPC#Reader gets a Call > - > > Key: HADOOP-18082 > URL: https://issues.apache.org/jira/browse/HADOOP-18082 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Now there is an important question. That is, we only know when a Call is > actually executed by the Handler. By logging, we cannot know when the Call > came in. > If I log some information from the moment the Call enters the inside of the > RPC, it will help us to know more about the Call. > The records here should be in the form of logs, and the priority of the logs > should not be too high, debug is the best. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18082) Add debug log when RPC#Reader gets a Call
[ https://issues.apache.org/jira/browse/HADOOP-18082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18082. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add debug log when RPC#Reader gets a Call > - > > Key: HADOOP-18082 > URL: https://issues.apache.org/jira/browse/HADOOP-18082 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Now there is an important question. That is, we only know when a Call is > actually executed by the Handler. By logging, we cannot know when the Call > came in. > If I log some information from the moment the Call enters the inside of the > RPC, it will help us to know more about the Call. > The records here should be in the form of logs, and the priority of the logs > should not be too high, debug is the best. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18096) Distcp: Sync moves filtered file to home directory rather than deleting
[ https://issues.apache.org/jira/browse/HADOOP-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18096. --- Fix Version/s: 3.4.0 3.3.2 Hadoop Flags: Reviewed Resolution: Fixed > Distcp: Sync moves filtered file to home directory rather than deleting > --- > > Key: HADOOP-18096 > URL: https://issues.apache.org/jira/browse/HADOOP-18096 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Distcp sync with snapshot, if the file being copied is renamed to a path > which is in the exclusion filter, tries to delete the file. > But instead of deleting, it moves the file to home directory > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-18024) SocketChannel is not closed when IOException happens in Server$Listener.doAccept
[
https://issues.apache.org/jira/browse/HADOOP-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reopened HADOOP-18024:
---
> SocketChannel is not closed when IOException happens in
> Server$Listener.doAccept
>
>
> Key: HADOOP-18024
> URL: https://issues.apache.org/jira/browse/HADOOP-18024
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.2.2
>Reporter: Haoze Wu
>Assignee: Haoze Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This is a follow-up of HADOOP-17552.
> When the symptom described in HADOOP-17552 happens, the client may time out
> in 2min, according to the default RPC timeout configuration specified in
> HADOOP-17552. Before this timeout, the client just waits, and does not know
> this issue happens.
> However, we recently found that actually the client doesn’t need to waste
> this 2min, and the server’s availability can be also improved. If the
> IOException happens in line 1402 or 1403 or 1404, we can just close this
> problematic `SocketChannel` and continue to accept new socket connections.
> The client side can also be aware of the close socket immediately, instead of
> waiting 2min.
> The old implementation:
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> public void run() {
> while (running) {
> // ...
> try {
> // ...
> while (iter.hasNext()) {
> // ...
> try {
> if (key.isValid()) {
> if (key.isAcceptable())
> doAccept(key); // line 1348
> }
> } catch (IOException e) { // line 1350
> }
> // ...
> }
> } catch (OutOfMemoryError e) {
> // ...
> } catch (Exception e) {
> // ...
> }
> }
> } {code}
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> Reader reader = getReader();
> Connection c = connectionManager.register(channel,
> this.listenPort, this.isOnAuxiliaryPort);
> // If the connectionManager can't take it, close the connection.
> if (c == null) {
> if (channel.isOpen()) {
> IOUtils.cleanup(null, channel);
> }
> connectionManager.droppedConnections.getAndIncrement();
> continue;
> }
> key.attach(c); // so closeCurrentConnection can get the object
> reader.addConnection(c);
> }
> } {code}
>
> We propose that the following implementation is better:
> {code:java}
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> try {
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> } catch (IOException e) {
> LOG.warn(...);
> try {
> channel.socket().close();
> channel.close();
> } catch (IOException ignored) { }
> continue;
> }
> // ...
> }
> }{code}
> The advantages include:
> # {*}In the old implementation{*}, the `ServerSocketChannel` was abandoned
> due to the single exception in this single `SocketChannel`, because the
> exception handler is in line 1350. {*}In the new implementation{*}, we use a
> try-catch to handle the exception in line 1402 or 1403 or 1404, then the
> `ServerSocketChannel` can continue to accept new connections, and don’t need
> to go back to the line 1348 in the next while loop in the run method.
> # {*}In the old implementation{*}, the client (another endpoint of this
> `SocketChannel`) is not aware of this
[jira] [Created] (HADOOP-18096) Distcp: Sync moves filtered file to home directory rather than deleting
Ayush Saxena created HADOOP-18096: - Summary: Distcp: Sync moves filtered file to home directory rather than deleting Key: HADOOP-18096 URL: https://issues.apache.org/jira/browse/HADOOP-18096 Project: Hadoop Common Issue Type: Bug Reporter: Ayush Saxena Assignee: Ayush Saxena Distcp sync with snapshot, if the file being copied is renamed to a path which is in the exclusion filter, tries to delete the file. But instead of deleting, it moves the file to home directory -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18093) Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java
[
https://issues.apache.org/jira/browse/HADOOP-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HADOOP-18093.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Better exception handling for testFileStatusOnMountLink() in
> ViewFsBaseTest.java
>
>
> Key: HADOOP-18093
> URL: https://issues.apache.org/jira/browse/HADOOP-18093
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The following exception handling code section in testFileStatusOnMountLink()
>
> {code:java}
> try {
> fcView.getFileStatus(new Path("/danglingLink"));
> Assert.fail("Excepted a not found exception here");
> } catch ( FileNotFoundException e) {
> // as excepted
> }{code}
> can be replaced with a single in the @Test
> {code:java}
> @Test(expected = FileNotFoundException.class){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18056) DistCp: Filter duplicates in the source paths
[ https://issues.apache.org/jira/browse/HADOOP-18056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18056. --- Fix Version/s: 3.4.0 3.3.3 Hadoop Flags: Reviewed Resolution: Fixed > DistCp: Filter duplicates in the source paths > - > > Key: HADOOP-18056 > URL: https://issues.apache.org/jira/browse/HADOOP-18056 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Add a basic filtering to remove the exact duplicate paths exposed for copying. > In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp > fails with DuplicateFileException, post building the listing. > Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18061) Update the year to 2022
[ https://issues.apache.org/jira/browse/HADOOP-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18061. --- Fix Version/s: 3.4.0 2.10.2 3.2.3 3.3.2 Hadoop Flags: Reviewed Resolution: Fixed > Update the year to 2022 > --- > > Key: HADOOP-18061 > URL: https://issues.apache.org/jira/browse/HADOOP-18061 > Project: Hadoop Common > Issue Type: Task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 40m > Remaining Estimate: 0h > > Update the year to 2022 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18061) Update the year to 2022
Ayush Saxena created HADOOP-18061: - Summary: Update the year to 2022 Key: HADOOP-18061 URL: https://issues.apache.org/jira/browse/HADOOP-18061 Project: Hadoop Common Issue Type: Task Reporter: Ayush Saxena Assignee: Ayush Saxena Update the year to 2022 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18056) DistCp: Filter duplicates in the source paths
Ayush Saxena created HADOOP-18056: - Summary: DistCp: Filter duplicates in the source paths Key: HADOOP-18056 URL: https://issues.apache.org/jira/browse/HADOOP-18056 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Assignee: Ayush Saxena Add a basic filtering to remove the exact duplicate paths exposed for copying. In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp fails with DuplicateFileException, post building the listing. Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18001) Update to Jetty 9.4.44
[ https://issues.apache.org/jira/browse/HADOOP-18001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18001. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update to Jetty 9.4.44 > -- > > Key: HADOOP-18001 > URL: https://issues.apache.org/jira/browse/HADOOP-18001 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yuan Luo >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HADOOP-18001.001.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18034) Bump mina-core from 2.0.16 to 2.1.5 in /hadoop-project
[ https://issues.apache.org/jira/browse/HADOOP-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18034. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Bump mina-core from 2.0.16 to 2.1.5 in /hadoop-project > --- > > Key: HADOOP-18034 > URL: https://issues.apache.org/jira/browse/HADOOP-18034 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Raised from github depandot -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18034) Bump mina-core from 2.0.16 to 2.1.5 in /hadoop-project
Ayush Saxena created HADOOP-18034: - Summary: Bump mina-core from 2.0.16 to 2.1.5 in /hadoop-project Key: HADOOP-18034 URL: https://issues.apache.org/jira/browse/HADOOP-18034 Project: Hadoop Common Issue Type: Improvement Reporter: Ayush Saxena Raised from github depandot -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18005) Correct log format for LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18005. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Correct log format for LdapGroupsMapping > > > Key: HADOOP-18005 > URL: https://issues.apache.org/jira/browse/HADOOP-18005 > Project: Hadoop Common > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-12-09-20-58-968.png, > image-2021-11-12-09-22-12-454.png > > Time Spent: 40m > Remaining Estimate: 0h > > Correct log format for LdapGroupsMapping. > !image-2021-11-12-09-22-12-454.png|width=557,height=101! > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17888) The error of Constant annotation in AzureNativeFileSystemStore.java
[ https://issues.apache.org/jira/browse/HADOOP-17888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-17888. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > The error of Constant annotation in AzureNativeFileSystemStore.java > > > Key: HADOOP-17888 > URL: https://issues.apache.org/jira/browse/HADOOP-17888 > Project: Hadoop Common > Issue Type: Improvement >Reporter: guoxin >Assignee: guoxin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17907) FileUtil#fullyDelete deletes contents of sym-linked directory when symlink cannot be deleted because of local fs fault
[ https://issues.apache.org/jira/browse/HADOOP-17907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-17907. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > FileUtil#fullyDelete deletes contents of sym-linked directory when symlink > cannot be deleted because of local fs fault > -- > > Key: HADOOP-17907 > URL: https://issues.apache.org/jira/browse/HADOOP-17907 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Weihao Zheng >Assignee: Weihao Zheng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > As discussed in HADOOP-6536, FileUtil#fullyDelete should not delete the > contents of the sym-linked directory when we pass a symlink parameter. > Currently we try to delete the resource first by calling deleteImpl, and if > deleteImpl is failed, we regard it as non-empty directory and remove all its > contents and then itself. This logic behaves wrong when local file system > cannot delete symlink to a directory because of faulty disk, local system's > error, etc. When we cannot delete it in the first time, hadoop will try to > remove all the contents of the directory it pointed to and leave an empty dir. > So, we should add an isSymlink checking before we call fullyDeleteContents to > prevent such behavior. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
