[jira] [Created] (HADOOP-16837) Spark-SQL test running on Windows: hadoop chgrp warnings
Guram Savinov created HADOOP-16837:
--
Summary: Spark-SQL test running on Windows: hadoop chgrp warnings
Key: HADOOP-16837
URL: https://issues.apache.org/jira/browse/HADOOP-16837
Project: Hadoop Common
Issue Type: Bug
Components: common, fs
Affects Versions: 2.6.5
Environment: Windows 10
Winutils 2.7.1:
[https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1]
Oracle JavaSE 8
SparkSQL 2.4.4 / Hadoop 2.6.5
Using: -Dhive.exec.scratchdir=C:\Users\OSUser\hadoop\tmp\hive
Set: winutils chmod -R 777 \Users\OSUser\hadoop\tmp\hive
Reporter: Guram Savinov
Attachments: HadoopGroupTest.java
Running SparkSQL local embedded unit tests on Win10, using winutils.
Got warnings about 'hadoop chgrp'.
See environment info.
{code:bash}
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
{code}
Related info on SO:
[https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive]
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210
The problem is: backslash character isn't included to allowedChars, see
attached HadoopGroupTest.java
Original issue in Spark: https://issues.apache.org/jira/browse/SPARK-30701
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11867) FS API: Add a high-performance vectored Read to FSDataInputStream API
[
https://issues.apache.org/jira/browse/HADOOP-11867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029389#comment-17029389
]
Owen O'Malley commented on HADOOP-11867:
Ok, my first patch adds the API, the default method, and the utilities for
this. I also included the implementation for RawLocalFileSystem and
ChecksumFileSystem because they were easiest to test and let me use the APIs in
non-trivial ways. I also included a benchmark that tests against the local file
system:
{code}
Benchmark (bufferKind) (fileSystemKind) Mode Cnt
Score Error Units
AsyncBenchmark.asyncFileChanArraydirect N/A avgt 20
1432.396 ± 232.443 us/op
AsyncBenchmark.asyncFileChanArray array N/A avgt 20
1551.400 ± 65.639 us/op
AsyncBenchmark.asyncRead direct local avgt 20
2514.926 ± 245.603 us/op
AsyncBenchmark.asyncRead direct raw avgt 20
1440.434 ± 207.504 us/op
AsyncBenchmark.asyncRead array local avgt 20
2798.031 ± 135.023 us/op
AsyncBenchmark.asyncRead array raw avgt 20
1524.360 ± 54.462 us/op
AsyncBenchmark.syncRead N/A local avgt 20
9515.604 ± 123.311 us/op
AsyncBenchmark.syncRead N/A raw avgt 20
2402.039 ± 36.620 us/op
{code}
> FS API: Add a high-performance vectored Read to FSDataInputStream API
> -
>
> Key: HADOOP-11867
> URL: https://issues.apache.org/jira/browse/HADOOP-11867
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, fs/s3, hdfs-client
>Affects Versions: 3.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Owen O'Malley
>Priority: Major
> Labels: performance
>
> The most significant way to read from a filesystem in an efficient way is to
> let the FileSystem implementation handle the seek behaviour underneath the
> API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read
> locations as part of a single call, while letting the system schedule/plan
> the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows
> for potentially optimizing away the seek-gaps within the FSDataInputStream
> implementation.
> For seek+read systems with even more latency than locally-attached disks,
> something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would
> take of the seeks internally while reading chunk.remaining() bytes into each
> chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into
> ByteBuffers, without forcing each FS implementation to override this in any
> way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley opened a new pull request #1830: HADOOP-11867: Add gather API to file system.
omalley opened a new pull request #1830: HADOOP-11867: Add gather API to file system. URL: https://github.com/apache/hadoop/pull/1830 Add API to PositionedReadable to have an asynchronous gather API. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move et
hadoop-yetus commented on issue #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... URL: https://github.com/apache/hadoop/pull/1829#issuecomment-581601343 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 21m 1s | trunk passed | | +1 :green_heart: | compile | 1m 4s | trunk passed | | +1 :green_heart: | checkstyle | 0m 49s | trunk passed | | +1 :green_heart: | mvnsite | 1m 11s | trunk passed | | +1 :green_heart: | shadedclient | 15m 51s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 22s | trunk passed | | +0 :ok: | spotbugs | 3m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 9s | trunk passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 32s | hadoop-hdfs in the patch failed. | | -1 :x: | compile | 0m 33s | hadoop-hdfs in the patch failed. | | -1 :x: | javac | 0m 33s | hadoop-hdfs in the patch failed. | | -0 :warning: | checkstyle | 0m 46s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 27 new + 245 unchanged - 0 fixed = 272 total (was 245) | | -1 :x: | mvnsite | 0m 36s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | shadedclient | 3m 46s | patch has errors when building and testing our client artifacts. | | -1 :x: | javadoc | 1m 14s | hadoop-hdfs-project_hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -1 :x: | findbugs | 0m 36s | hadoop-hdfs in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 36s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 55m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1829 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6ed8ff3e7720 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1e3a0b0 | | Default Java | 1.8.0_242 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/testReport/ | | Max. process+thread count | 303 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1829/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated.
[jira] [Commented] (HADOOP-16438) Introduce a config to control SSL Channel mode in Azure DataLake Store Gen1
[ https://issues.apache.org/jira/browse/HADOOP-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029218#comment-17029218 ] Steve Loughran commented on HADOOP-16438: - Looks like a side effect of this is that it removes a copy of the wildfly 1.0.4 classes from the ADLS JAR. This is needed to reliably get the upgrade to wildfly 1.0.7 on the CP. tagging as a dependency for HADOOP-16460 > Introduce a config to control SSL Channel mode in Azure DataLake Store Gen1 > --- > > Key: HADOOP-16438 > URL: https://issues.apache.org/jira/browse/HADOOP-16438 > Project: Hadoop Common > Issue Type: Task > Components: fs/adl >Affects Versions: 2.9.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16438.001.patch > > > Currently there is no user control possible on the SSL channel mode used for > server connections. It will try to connect using SSLChannelMode.OpenSSL and > default to SSLChannelMode.Default_JSE when there is any issue. > A new config is needed to toggle the choice if any issues are observed with > OpenSSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16438) Introduce a config to control SSL Channel mode in Azure DataLake Store Gen1
[ https://issues.apache.org/jira/browse/HADOOP-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16438: Priority: Major (was: Minor) > Introduce a config to control SSL Channel mode in Azure DataLake Store Gen1 > --- > > Key: HADOOP-16438 > URL: https://issues.apache.org/jira/browse/HADOOP-16438 > Project: Hadoop Common > Issue Type: Task > Components: fs/adl >Affects Versions: 2.9.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16438.001.patch > > > Currently there is no user control possible on the SSL channel mode used for > server connections. It will try to connect using SSLChannelMode.OpenSSL and > default to SSLChannelMode.Default_JSE when there is any issue. > A new config is needed to toggle the choice if any issues are observed with > OpenSSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, mo
jojochuang opened a new pull request #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... URL: https://github.com/apache/hadoop/pull/1829 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029149#comment-17029149
]
Hadoop QA commented on HADOOP-16833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HADOOP-16833 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992512/HADOOP-16833.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8af0735e8012 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e3a0b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16751/testReport/ |
| Max. process+thread count | 1376 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16751/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL:
[jira] [Commented] (HADOOP-15643) Review/implement ABFS support for the extra fs ops which some apps (HBase) expects
[ https://issues.apache.org/jira/browse/HADOOP-15643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029085#comment-17029085 ] Anoop Sam John commented on HADOOP-15643: - HBase now having ByteBufferReadable usage also. > Review/implement ABFS support for the extra fs ops which some apps (HBase) > expects > -- > > Key: HADOOP-15643 > URL: https://issues.apache.org/jira/browse/HADOOP-15643 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Priority: Major > > One troublespot with storage connectors is those apps which expect rarer > APIs, e.g. Beam and ByteBufferReadable ( BEAM-2790), HBase and CanUnbuffer > (HADOOP-14748). > Review ABFS support with these, decide which to implement, and if not, make > sure that the callers can handle that -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1822: HADOOP-16832 Update S3Guard testing doc.
hadoop-yetus commented on issue #1822: HADOOP-16832 Update S3Guard testing doc. URL: https://github.com/apache/hadoop/pull/1822#issuecomment-581492400 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 25m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 37s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 32m 41s | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 25s | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 74m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1822/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1822 | | Optional Tests | dupname asflicense mvnsite markdownlint | | uname | Linux 1185ad913813 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1e3a0b0 | | Max. process+thread count | 412 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1822/3/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HADOOP-16833:
---
Attachment: HADOOP-16833.004.patch
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch, HADOOP-16833.004.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.
hadoop-yetus commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation. URL: https://github.com/apache/hadoop/pull/1823#issuecomment-581467919 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 13s | trunk passed | | +1 :green_heart: | compile | 0m 35s | trunk passed | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 39s | trunk passed | | +1 :green_heart: | shadedclient | 13m 29s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | trunk passed | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 57s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | hadoop-tools/hadoop-aws: The patch generated 0 new + 13 unchanged - 1 fixed = 13 total (was 14) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 20s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | | +1 :green_heart: | findbugs | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 55m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1823 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 48283c2c2842 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1e3a0b0 | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/4/testReport/ | | Max. process+thread count | 454 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029015#comment-17029015
]
Hadoop QA commented on HADOOP-16819:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Inconsistent synchronization of
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.currentKey;
locked 88% of time Unsynchronized access at
AbstractDelegationTokenSecretManager.java:88% of time Unsynchronized access at
AbstractDelegationTokenSecretManager.java:[line 366] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HADOOP-16819 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991450/HADOOP-16819.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 88019b7a42cf 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e3a0b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
|
[GitHub] [hadoop] hadoop-yetus commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.
hadoop-yetus commented on issue #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation. URL: https://github.com/apache/hadoop/pull/1823#issuecomment-581439423 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 30m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 14s | trunk passed | | +1 :green_heart: | compile | 0m 35s | trunk passed | | +1 :green_heart: | checkstyle | 0m 28s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 13m 30s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14) | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 20s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | | +1 :green_heart: | findbugs | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 25s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 85m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1823 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 69568f12de59 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1e3a0b0 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/3/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1823/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028979#comment-17028979
]
Hadoop QA commented on HADOOP-16833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HADOOP-16833 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992489/HADOOP-16833.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 07400c9d6f36 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e3a0b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16749/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16749/testReport/ |
| Max. process+thread count | 1795 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16749/console |
| Powered by | Apache Yetus
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client. URL: https://github.com/apache/hadoop/pull/1826#issuecomment-581431289 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 18m 0s | trunk passed | | +1 :green_heart: | compile | 16m 35s | trunk passed | | +1 :green_heart: | checkstyle | 2m 38s | trunk passed | | +1 :green_heart: | mvnsite | 2m 17s | trunk passed | | +1 :green_heart: | shadedclient | 18m 44s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 13s | trunk passed | | +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 17s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 22s | the patch passed | | +1 :green_heart: | compile | 15m 54s | the patch passed | | +1 :green_heart: | javac | 15m 54s | the patch passed | | -0 :warning: | checkstyle | 2m 37s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) | | +1 :green_heart: | mvnsite | 2m 19s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 12m 28s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 17s | the patch passed | | +1 :green_heart: | findbugs | 3m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 5s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | | | | 118m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1826 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 95e07a7275de 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1e3a0b0 | | Default Java | 1.8.0_232 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/diff-checkstyle-root.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/testReport/ | | Max. process+thread count | 1594 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.
steveloughran commented on a change in pull request #1823: HADOOP-16794 S3
Encryption key is not getting set properly during put operation.
URL: https://github.com/apache/hadoop/pull/1823#discussion_r374113961
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionWithDefaultS3Settings.java
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.IOException;
+
+import com.amazonaws.services.s3.model.ObjectMetadata;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.skip;
+import static org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_KEY;
+import static org.apache.hadoop.fs.s3a.S3AEncryptionMethods.SSE_KMS;
+
+/**
+ * Concrete class that extends {@link AbstractTestS3AEncryption}
+ * and tests already configured bucket level encryption using s3 console.
+ * This requires the SERVER_SIDE_ENCRYPTION_KEY
+ * to be set in auth-keys.xml for it to run. The value should match with the
+ * kms key set for the bucket.
+ */
+public class ITestS3AEncryptionWithDefaultS3Settings extends
+AbstractTestS3AEncryption {
+
+ @Override
+ protected Configuration getConfiguration() {
+// get the KMS key for this test.
+Configuration c = new Configuration();
+String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
+if (StringUtils.isBlank(kmsKey)) {
+ skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
+ SSE_KMS.getMethod());
+}
+return super.createConfiguration();
+ }
+
+ /**
+ * Setting this to NONE as we don't want to overwrite
+ * already configured encryption settings.
+ * @return
+ */
+ @Override
+ protected S3AEncryptionMethods getSSEAlgorithm() {
+return S3AEncryptionMethods.NONE;
+ }
+
+ @Override
+ protected void assertEncrypted(Path path) throws IOException {
+Configuration c = new Configuration();
+String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
+ObjectMetadata objectMetadata = getFileSystem().getObjectMetadata(path);
+assertEquals("SSE KMS key id should match", kmsKey,
objectMetadata.getSSEAwsKmsKeyId());
+ }
+
+ @Override
+ @Ignore
+ @Test
+ public void testEncryptionSettingPropagation() throws Throwable {
+super.testEncryptionSettingPropagation();
+ }
+
+ @Override
+ @Ignore
+ @Test
+ public void testEncryption() throws Throwable {
+super.testEncryption();
Review comment:
not needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.
steveloughran commented on a change in pull request #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation. URL: https://github.com/apache/hadoop/pull/1823#discussion_r374113472 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionWithDefaultS3Settings.java ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import java.io.IOException; + +import com.amazonaws.services.s3.model.ObjectMetadata; + +import org.apache.commons.lang3.StringUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.junit.Ignore; Review comment: should go into the same block as the amazon one This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1823: HADOOP-16794 S3 Encryption key is not getting set properly during put operation.
steveloughran commented on a change in pull request #1823: HADOOP-16794 S3
Encryption key is not getting set properly during put operation.
URL: https://github.com/apache/hadoop/pull/1823#discussion_r374114951
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionWithDefaultS3Settings.java
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.io.IOException;
+
+import com.amazonaws.services.s3.model.ObjectMetadata;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.junit.Ignore;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.skip;
+import static org.apache.hadoop.fs.s3a.Constants.SERVER_SIDE_ENCRYPTION_KEY;
+import static org.apache.hadoop.fs.s3a.S3AEncryptionMethods.SSE_KMS;
+
+/**
+ * Concrete class that extends {@link AbstractTestS3AEncryption}
+ * and tests already configured bucket level encryption using s3 console.
+ * This requires the SERVER_SIDE_ENCRYPTION_KEY
+ * to be set in auth-keys.xml for it to run. The value should match with the
+ * kms key set for the bucket.
+ */
+public class ITestS3AEncryptionWithDefaultS3Settings extends
+AbstractTestS3AEncryption {
+
+ @Override
+ protected Configuration getConfiguration() {
+// get the KMS key for this test.
+Configuration c = new Configuration();
+String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
+if (StringUtils.isBlank(kmsKey)) {
+ skip(SERVER_SIDE_ENCRYPTION_KEY + " is not set for " +
+ SSE_KMS.getMethod());
+}
+return super.createConfiguration();
+ }
+
+ /**
+ * Setting this to NONE as we don't want to overwrite
+ * already configured encryption settings.
+ * @return
+ */
+ @Override
+ protected S3AEncryptionMethods getSSEAlgorithm() {
+return S3AEncryptionMethods.NONE;
+ }
+
+ @Override
+ protected void assertEncrypted(Path path) throws IOException {
+Configuration c = new Configuration();
+String kmsKey = c.get(SERVER_SIDE_ENCRYPTION_KEY);
+ObjectMetadata objectMetadata = getFileSystem().getObjectMetadata(path);
+assertEquals("SSE KMS key id should match", kmsKey,
objectMetadata.getSSEAwsKmsKeyId());
Review comment:
see checkstyle -line needs splitting. We like under/near 80 chars for better
side-by-side review
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16819:
Affects Version/s: 3.3.0
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, security
>Affects Versions: 3.3.0
>Reporter: Hankó Gergely
>Priority: Major
> Attachments: HADOOP-16819.001.patch
>
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16819:
Parent: HADOOP-15620
Issue Type: Sub-task (was: Bug)
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Reporter: Hankó Gergely
>Priority: Major
> Attachments: HADOOP-16819.001.patch
>
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16819:
Component/s: fs/s3
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, security
>Reporter: Hankó Gergely
>Priority: Major
> Attachments: HADOOP-16819.001.patch
>
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1822: HADOOP-16832 Update S3Guard testing doc.
steveloughran commented on issue #1822: HADOOP-16832 Update S3Guard testing doc. URL: https://github.com/apache/hadoop/pull/1822#issuecomment-581410993 It needs a sentence at the beginning: How to set test options in your hadoop configuration rather than on the maven command line. And at the end *warning* although easier for IDE debugging setups, once you do this, you cannot change configurations on the mvn command line, such as testing without s3guard. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028920#comment-17028920
]
Stephen O'Donnell edited comment on HADOOP-16833 at 2/3/20 1:12 PM:
I uploaded a v3 patch. This ensure the instrumented lock will track the maximum
wait time it suppressed.
The way the DN uses this lock, it sets a suppression interval of 10 seconds,
and a log threshold of 300ms.
The the first thread which exceeds the threshold and there has been at least 10
seconds since the last log message will get logged. But its possible for
another thread to have waited longer and the suppressed message does not give
any clues about the longest wait time, and it would be nice to see that.
Eg
{code}
-> | Log Message --| Log Message (triggered by "short
wait") >
| Waiting ---> | suppressed
| short Wait -> | (this is triggers another log,
masking the long wait)
{code}
I am not sure how clear that diagram makes it, but basically a tread could have
been waiting for some time and the short waiter gets logged. With this change
we should also get the longest waiting time during that interval.
Another idea I considered, is whether the log wait threshold should be a
multiple of the write hold interval. Eg log a lock held over 300ms, but a lock
wait of 600ms? Its hard to know if this would be useful or not without running
with this patch in production for some time to see how many times we get log
waits logged, as right now we have no information around that. TBH, if we are
regularly seeing lock waits over 300ms, there is likely to be a problem, so
perhaps keeping both thresholds the same is fine. I would welcome some thoughts
on this.
was (Author: sodonnell):
I uploaded a v3 patch. This ensure the instrumented lock will track the maximum
wait time it suppressed.
The way the DN uses this lock, it sets a suppression interval of 10 seconds,
and a log threshold of 300ms.
The the first thread which exceeds the threshold and there has been at least 10
seconds since the last log message will get logged. But its possible for
another thread to have waited longer and the suppressed message does not give
any clues about the longest wait time, and it would be nice to see that.
Eg
{code}
-> | Log Message --| Log Message (triggered by "short
wait") >
| Waiting ---> | suppressed
| short Wait -> | (this is triggers another log,
masking the long wait)
{code}
I am not sure how clear that diagram makes it, but basically a tread could have
been waiting for some time and the short waiter gets logged. With this change
we should also get the longest waiting time during that interval.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028920#comment-17028920
]
Stephen O'Donnell edited comment on HADOOP-16833 at 2/3/20 1:08 PM:
I uploaded a v3 patch. This ensure the instrumented lock will track the maximum
wait time it suppressed.
The way the DN uses this lock, it sets a suppression interval of 10 seconds,
and a log threshold of 300ms.
The the first thread which exceeds the threshold and there has been at least 10
seconds since the last log message will get logged. But its possible for
another thread to have waited longer and the suppressed message does not give
any clues about the longest wait time, and it would be nice to see that.
Eg
{code}
-> | Log Message --| Log Message (triggered by "short
wait") >
| Waiting ---> | suppressed
| short Wait -> | (this is triggers another log,
masking the long wait)
{code}
I am not sure how clear that diagram makes it, but basically a tread could have
been waiting for some time and the short waiter gets logged. With this change
we should also get the longest waiting time during that interval.
was (Author: sodonnell):
I uploaded a v3 patch. This ensure the instrumented lock will track the maximum
wait time it suppressed.
The way the DN uses this lock, it sets a suppression interval of 10 seconds,
and a log threshold of 300ms.
The the first thread which exceeds the threshold and there has been at least 10
seconds since the last log message will get logged. But its possible for
another thread to have waited longer and the suppressed message does not give
any clues about the longest wait time, and it would be nice to see that.
Eg
{code}
-> | Log Message --| Log Message (triggered by "short
wait") >
| Waiting -> | suppressed
| short Wait -> | (this is
triggers another log, masking the long wait)
{code}
I am not sure how clear that diagram makes it, but basically a tread could have
been waiting for some time and the short waiter gets logged. With this change
we should also get the longest waiting time during that interval.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028920#comment-17028920
]
Stephen O'Donnell commented on HADOOP-16833:
I uploaded a v3 patch. This ensure the instrumented lock will track the maximum
wait time it suppressed.
The way the DN uses this lock, it sets a suppression interval of 10 seconds,
and a log threshold of 300ms.
The the first thread which exceeds the threshold and there has been at least 10
seconds since the last log message will get logged. But its possible for
another thread to have waited longer and the suppressed message does not give
any clues about the longest wait time, and it would be nice to see that.
Eg
{code}
-> | Log Message --| Log Message (triggered by "short
wait") >
| Waiting -> | suppressed
| short Wait -> | (this is
triggers another log, masking the long wait)
{code}
I am not sure how clear that diagram makes it, but basically a tread could have
been waiting for some time and the short waiter gets logged. With this change
we should also get the longest waiting time during that interval.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HADOOP-16833:
---
Attachment: HADOOP-16833.003.patch
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
