Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/530/ [Aug 1, 2023, 9:52:54 AM] (github) HDFS-17134. RBF: Fix duplicate results of getListing through Router. (#5900). Contributed by Shuyan Zhang. [Aug 2, 2023, 6:10:24 PM] (github) YARN-3660. [Addendum] Fix GPG Pom.xml Typo, (#5903) -1 overall The following subsystems voted -1: blanks hadolint mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState)) Redundant null check at ResourceLocalizationService.java:is known to be non-null in
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1108/ No changes ERROR: File 'out/email-report.txt' does not exist - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/ No changes -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.tools.TestDFSAdmin hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.balancer.TestBalancer cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-compile-javac-root.txt [380K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/xml.txt [28K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/results-javadoc-javadoc-root.txt [972K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [264K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/120/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [544K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/ [Aug 2, 2023, 6:10:24 PM] (github) YARN-3660. [Addendum] Fix GPG Pom.xml Typo, (#5903) -1 overall The following subsystems voted -1: blanks hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.net.TestSocketIOWithTimeout hadoop.hdfs.server.namenode.ha.TestObserverNode hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.mapreduce.v2.TestUberAM hadoop.mapreduce.v2.TestMRJobsWithProfiler hadoop.mapreduce.v2.TestMRJobs cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-hadolint.txt [20K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/results-javadoc-javadoc-root.txt [244K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [32K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1307/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [72K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18839) SSLException while accessing S3 bucket is reported only after 15 minutes of waiting
Maxim Martynov created HADOOP-18839:
---
Summary: SSLException while accessing S3 bucket is reported only
after 15 minutes of waiting
Key: HADOOP-18839
URL: https://issues.apache.org/jira/browse/HADOOP-18839
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.4
Reporter: Maxim Martynov
Attachments: host.log, ssl.log
I've tried to connect from PySpark to Minio running in docker.
Installing PySpark and starting Minio:
{code:bash}
pip install pyspark==3.4.1
docker run --rm -d --hostname minio --name minio -p 9000:9000 -p 9001:9001 -e
MINIO_ACCESS_KEY=access -e MINIO_SECRET_KEY=Eevoh2wo0ui6ech0wu8oy
3feiR3eicha -e MINIO_ROOT_USER=admin -e
MINIO_ROOT_PASSWORD=iepaegaigi3ofa9TaephieSo1iecaesh bitnami/minio:latest
docker exec minio mc mb test-bucket
{code}
Then create Spark session:
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder\
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4")\
.config("spark.hadoop.fs.s3a.endpoint", "localhost:9000")\
.config("spark.hadoop.fs.s3a.access.key", "access")\
.config("spark.hadoop.fs.s3a.secret.key",
"Eevoh2wo0ui6ech0wu8oy3feiR3eicha")\
.config("spark.hadoop.fs.s3a.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")\
.getOrCreate()
spark.sparkContext.setLogLevel("debug")
{code}
And try to access some object in a bucket:
{code:python}
import time
begin = time.perf_counter()
spark.read.format("csv").load("s3a://test-bucket/fake")
end = time.perf_counter()
py4j.protocol.Py4JJavaError: An error occurred while calling o40.load.
: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
HTTP request: Unsupported or unrecognized SSL message: Unable to execute HTTP
request: Unsupported or unrecognized SSL message
...
{code}
[^ssl.log]
{code:python}
>>> print((end-begin)/60)
14.72387898775002
{code}
I was waiting almost *15 minutes* to get the exception from Spark. The reason
was I tried to connect to S3 instance with
{{{}fs.s3a.connection.ssl.enabled=true{}}}, but Minio is configured to listen
for HTTP protocol only.
Is there any way to immediately raise exception if SSL connection cannot be
established?
If I try to pass wrong endpoint, like {{{}localhos:9000{}}}, I'll get exception
like this in just 5 seconds:
{code:java}
: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
s3a://test-bucket/fake: com.amazonaws.SdkClientException: Unable to execute
HTTP request: test-bucket.localhos: Unable to execute HTTP request:
test-bucket.localhos
...
{code}
[^host.log]
{code:python}
>>> print((end-begin)/60)
0.09500707178334172
>>> end-begin
5.700424307000503
{code}
I know about options like {{fs.s3a.attempts.maximum}} and
{{{}fs.s3a.retry.limit{}}}, setting them to 1 will cause raising exception just
immediately. But this does not look right.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18838) Some fs.s3a.* config values are different in sources and documentation
Maxim Martynov created HADOOP-18838:
---
Summary: Some fs.s3a.* config values are different in sources and
documentation
Key: HADOOP-18838
URL: https://issues.apache.org/jira/browse/HADOOP-18838
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.6
Reporter: Maxim Martynov
For config option {{fs.s3a.retry.throttle.interval}} default value in source
code is {{500ms}}:
{code:java}
public static final String RETRY_THROTTLE_INTERVAL_DEFAULT = "500ms";
{code}
https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L921
In {{core-default.xml}} it has value {{100ms}}:
{code:xml}
fs.s3a.retry.throttle.interval
100ms
Initial between retry attempts on throttled requests, +/- 50%. chosen at
random.
i.e. for an intial value of 3000ms, the initial delay would be in the range
1500ms to 4500ms.
Backoffs are exponential; again randomness is used to avoid the thundering
heard problem.
500ms is the default value used by the AWS S3 Retry policy.
{code}
https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1750
This change introduced in HADOOP-16823.
In Hadoop-AWS module index it has value {{1000ms}}:
{code:xml}
fs.s3a.retry.throttle.interval
1000ms
Interval between retry attempts on throttled requests.
{code}
https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md?plain=1#L1223
File was created in HADOOP-13786, and value is left unchanged since when.
In performance tuning page it has up-to-date value {{500ms}}:
{code:xml}
fs.s3a.retry.throttle.interval
500ms
Interval between retry attempts on throttled requests.
{code}
https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md?plain=1#L435
The same issue with:
* {{fs.s3a.retry.throttle.limit}} - in source code it has value {{20}}, but in
some documents still old value ${fs.s3a.attempts.maximum}
* {{fs.s3a.connection.establish.timeout}} - in source code it has value
{{50_000}}, in config file & documentation {{5_000}}
* {{fs.s3a.attempts.maximum}} - in source code it has value {{10}}, in config
file & documentation {{20}}
* {{fs.s3a.threads.max} - in source & documentation code it has value {{10}},
in config file {{64}}
* {{fs.s3a.max.total.tasks}} - in source code & config it has value {{32}}, in
documentation {{5}}
* {{fs.s3a.connection.maximum}} - in source code & config it has value {{96}},
in documentation {{15}} or {{30}}
Please sync these values, outdated documentation is very painful to work with.
As an idea, is it possible to use {{core-default.xml}} directly in
documentation, or generate this documentation from docstrings in Java code?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
