date:20230314

Hadoop depencency CVEs

2023-03-14 Thread Michiel de Jong


Hello Hadoop Developers,

When running a dependency cve scan on our project we noticed a list of 
dependencies in hadoop common that have some CVE. There are also several 
CVEs listed on 
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.4.
Many of these CVEs would probably not affect end users, however this is 
often difficult to determine for the end users themselves.


Is there a procedure in place for handling reported CVEs? Is there a 
place where the CVEs that do not impact end users are documented?


We would like to work on reducing the number of CVEs encountered in 
dependencies and document the CVEs that are not easily resolved and 
don't impact the end users.


Michiel

email signature

 * Winner of Dutch Innovation award within Law Enforcement
 * Active in 30+ countries

Michiel de Jong
Software Engineer

site:
PGP:web-iq.com 
5E01 D729 326D F933 4A20 C8CF 7D09 6113 7CFD 29DA

The content of this email is confidential and intended for the recipient 
specified in message only. It is strictly forbidden to share any part of 
this message with any third party, without a written consent of the 
sender. If you received this message by mistake, please reply to this 
message and follow with its deletion, so that we can ensure such a 
mistake does not occur in the future.




OpenPGP_0x7D0961137CFD29DA_and_old_rev.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[jira] [Created] (HADOOP-18663) DecayRpcSchedulerDetailedMetrics display name numbers incremented by 1

2023-03-14 Thread Christos Bisias (Jira)

Christos Bisias created HADOOP-18663:


 Summary: DecayRpcSchedulerDetailedMetrics display name numbers 
incremented by 1
 Key: HADOOP-18663
 URL: https://issues.apache.org/jira/browse/HADOOP-18663
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Christos Bisias


It should keep names consistent with 
{color:#00875a}DecayRpcSchedulerMetrics{color}, to avoid confusion.


What might be displayed as Priority.0 and Priority.1 for 
{color:#00875a}DecayRpcSchedulerMetrics{color}, it's Priority.1 and Priority.2 
for {color:#00875a}DecayRpcSchedulerDetailedMetrics{color}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18662) ListFiles with recursive fails with FNF

2023-03-14 Thread Ayush Saxena (Jira)

Ayush Saxena created HADOOP-18662:
-

 Summary: ListFiles with recursive fails with FNF
 Key: HADOOP-18662
 URL: https://issues.apache.org/jira/browse/HADOOP-18662
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena


Problem triggers in HDFS, but the change is in Hadoop-Common, Since the 
listFiles is defined in Hadoop-Common.

Scenario:

ListFiles With recursive: 
 * Fetches a dir say /dir, which has some /dir/s1...s10
 * Recursive is set to true: It goes and tries on say /dir/s5 and /dir/s5 got 
deleted by that time
 * The entire operation fails with FNF

Hive Cleaner uses listFiles with recursive true and this impacts that
{noformat}
2023-03-06 07:45:48,331 ERROR org.apache.hadoop.hive.ql.txn.compactor.Cleaner: 
[Cleaner-executor-thread-12]: Caught exception when cleaning, unable to 
complete cleaning of 
id:39762523,dbname:test,tableName:test_table,partName:null,state:,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:989,errorMessage:null,workerId:
 null,initiatorId: null java.io.FileNotFoundException: File 
hdfs:/cluster/warehouse/tablespace/managed/hive/test.db/test_table/.hive-staging_hive_2023-03-06_07-45-23_120_4659605113266849995-73550
 does not exist.
    at 
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1275)
    at 
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1249)
    at 
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
    at 
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
    at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at 
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
    at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
    at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:2332)
    at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:2309)
    at 
org.apache.hadoop.util.functional.RemoteIterators$WrappingRemoteIterator.sourceHasNext(RemoteIterators.java:432)
    at 
org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.fetch(RemoteIterators.java:581)
    at 
org.apache.hadoop.util.functional.RemoteIterators$FilteringRemoteIterator.hasNext(RemoteIterators.java:602)
    at 
org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1435)
    at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:287)
    at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:214)
    at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:114)
    at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
    at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2023-03-14 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/

[Mar 13, 2023, 4:38:04 AM] (github) HADOOP-18658. snakeyaml dependency: upgrade 
to v2.0 (#5467). Contributed by PJ Fanning.
[Mar 13, 2023, 12:24:36 PM] (Steve Loughran) HADOOP-18661. Fix bin/hadoop usage 
script terminology. (#5473)
[Mar 13, 2023, 12:30:12 PM] (github) HADOOP-18653. LogLevel servlet to 
determine log impl before using setLevel (#5456)




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-mapreduce-project/hadoop-mapreduce-client 
   Write to static field 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:[line 120] 

spotbugs :

   
module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
 
   Write to static field 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:[line 120] 

spotbugs :

   module:hadoop-mapreduce-project 
   Write to static field 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:[line 120] 

spotbugs :

   module:root 
   Write to static field 
org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:from instance method new 
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, 
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, 
ExceptionReporter, SecretKey) At Fetcher.java:[line 120] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/artifact/out/results-compile-javac-root.txt
 [528K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1165/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

[jira] [Created] (HADOOP-18665) IOUtils.wrapWithMessage can't wrap exceptions without string constructor

2023-03-14 Thread Steve Loughran (Jira)

Steve Loughran created HADOOP-18665:
---

 Summary: IOUtils.wrapWithMessage can't wrap exceptions without 
string constructor
 Key: HADOOP-18665
 URL: https://issues.apache.org/jira/browse/HADOOP-18665
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/azure, util
Affects Versions: 3.3.4
Reporter: Steve Loughran


When and attempt is made to wrap {{AbfsRestOperationException}}; it fails as 
there's no string constructor there.


{code}
java.lang.NoSuchMethodException: 
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException.(java.lang.String)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.hadoop.io.IOUtils.wrapWithMessage(IOUtils.java:507)
at org.apache.hadoop.io.IOUtils.wrapException(IOUtils.java:491)
at 
org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.close(AbfsOutputStream.java:495)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

{code}

proposed: handle NoSuchMethodException and downgrade to wrapping with a simple 
iOE. reluctant to not wrap in close() as we've had bizarre failures in the past 
there with try-with-resources failures




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18664) you can't launch create-release --docker from a build file

2023-03-14 Thread Steve Loughran (Jira)

Steve Loughran created HADOOP-18664:
---

 Summary: you can't launch create-release --docker from a build file
 Key: HADOOP-18664
 URL: https://issues.apache.org/jira/browse/HADOOP-18664
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.5
Reporter: Steve Loughran


the {{create-release}} script launches docker with -i for interactivity, which
breaks when launching it from an ant build file or other scripts


{code}
[x] 
[x] Failed!
[x] 
[x] the input device is not a TTY
[x] 3.12 real 0.29 user 0.39 sys

{code}

Fix: don't do that




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2023-03-14 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/966/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop depencency CVEs

2023-03-14 Thread Steve Loughran

hello.

welcome to the hadoop CVE support team!

all this stuff happens on apache JIRA; the search term is
project in (HADOOP, YARN, HDFS, MAPREDUCE) AND text ~ cve ORDER BY created
DESC

And we are cutting the 3.3.5 RC3 today; I just need to do the preflight
checks before sending the emails.
in the hadoop github repo, branch-3.3.5 is the one this is built off

please use that for your audits, not 3.3.4, and when the RC goes up, do as
much regression testing as you can

On Tue, 14 Mar 2023 at 08:27, Michiel de Jong 
wrote:

> Hello Hadoop Developers,
>
> When running a dependency cve scan on our project we noticed a list of
> dependencies in hadoop common that have some CVE. There are also several
> CVEs listed on https://mvnrepository.the
> com/artifact/org.apache.hadoop/hadoop-common/3.3.4
> 
> .
> Many of these CVEs would probably not affect end users, however this is
> often difficult to determine for the end users themselves.
>
> Is there a procedure in place for handling reported CVEs? Is there a place
> where the CVEs that do not impact end users are documented?
>
> We would like to work on reducing the number of CVEs encountered in
> dependencies and document the CVEs that are not easily resolved and don't
> impact the end users.
>
>
> where we are behind is  the javascript stuff -that's the YARN project; i
think it is undermaintained.

we also have to deal with the challenge of compatibility, especially a few
applications away. for example. this pair of commits reflects how an
upgrade broke hive/tez downstream.

HADOOP-18178. Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2
HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7

There's also the problem where libraries which generate classes (avro,
parquet) are brittle to library updates...if we update them then
applications which generated classes using the same lib just won't link any
more. There we are resorting to the hadoop shaded jar and trying to cut the
originals. though that adds more homework: keeping the shaded stuff current
given its extra homework with more release overhead

hadoop 3.3.5 isn't doing this, I'd be happy for someone to take up and
complete two PRs

HADOOP-18487. protobuf 2.5.0 marked as provided.
https://github.com/apache/hadoop/pull/4996

HADOOP-18197. Upgrade protobuf to 3.21.7
https://github.com/apache/hadoop-thirdparty/pull/19

This and any other CVE work can target the next release.

I am not going to hold back the 3.3.5 release for any more CVEs...we do
that and the following week's CVEs become blockers instead. HDFS critical
issues are the last bits of trouble.

steve

Hadoop depencency CVEs

[jira] [Created] (HADOOP-18663) DecayRpcSchedulerDetailedMetrics display name numbers incremented by 1

[jira] [Created] (HADOOP-18662) ListFiles with recursive fails with FNF

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[jira] [Created] (HADOOP-18665) IOUtils.wrapWithMessage can't wrap exceptions without string constructor

[jira] [Created] (HADOOP-18664) you can't launch create-release --docker from a build file

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

Re: Hadoop depencency CVEs

8 matches

Site Navigation

Mail list logo

Footer information