[jira] [Updated] (HDDS-797) If DN is started before SCM, it does not register
[ https://issues.apache.org/jira/browse/HDDS-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-797: Attachment: HDDS-797.001.patch > If DN is started before SCM, it does not register > - > > Key: HDDS-797 > URL: https://issues.apache.org/jira/browse/HDDS-797 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: Hanisha Koneru >Priority: Blocker > Attachments: HDDS-797.001.patch > > > If a DN is started before SCM, it does not register with the SCM. DNs keep > trying to connect with the SCM and once SCM is up, the DN services are > shutdown instead of registering with SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-798) Storage-class is showing incorrectly
[
https://issues.apache.org/jira/browse/HDDS-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-798:
Issue Type: Sub-task (was: Bug)
Parent: HDDS-763
> Storage-class is showing incorrectly
>
>
> Key: HDDS-798
> URL: https://issues.apache.org/jira/browse/HDDS-798
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>
> After HDDS-712, we support storage-class.
> For list-objects, even if key has set storage-class to REDUCED_REDUNDANCY,
> still it shows STANDARD.
> As in code in list object response, we have hardcoded it as below.
> keyMetadata.setStorageClass("STANDARD");
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-798) Storage-class is showing incorrectly
Bharat Viswanadham created HDDS-798:
---
Summary: Storage-class is showing incorrectly
Key: HDDS-798
URL: https://issues.apache.org/jira/browse/HDDS-798
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
After HDDS-712, we support storage-class.
For list-objects, even if key has set storage-class to REDUCED_REDUNDANCY,
still it shows STANDARD.
As in code in list object response, we have hardcoded it as below.
keyMetadata.setStorageClass("STANDARD");
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13834:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-13891
Status: Resolved (was: Patch Available)
Thanks [~crh] for the fix.
Committed to HDFS-13891.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Fix For: HDFS-13891
>
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834-HDFS-13891.5.patch, HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-658) Implement s3 bucket list backend call and use it from rest endpoint
[
https://issues.apache.org/jira/browse/HDDS-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673739#comment-16673739
]
Hadoop QA commented on HDDS-658:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 46s{color} | {color:orange} root: The patch generated 3 new + 0 unchanged -
0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} s3gateway in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
7s{color} | {color:green} integration-test in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color}
[jira] [Commented] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[
https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673756#comment-16673756
]
Hadoop QA commented on HDFS-14015:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
38m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs-native-client in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 48s{color} |
{color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 47s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14015 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946738/HDFS-14015.007.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 3109bb96d105 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aed836e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| cc |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/testReport/ |
| Max. process+thread count | 334 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25423/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error handling in hdfsThreadDestructor in native thread local storage
>
[jira] [Updated] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[ https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-14015: Attachment: HDFS-14015.008.patch > Improve error handling in hdfsThreadDestructor in native thread local storage > - > > Key: HDFS-14015 > URL: https://issues.apache.org/jira/browse/HDFS-14015 > Project: Hadoop HDFS > Issue Type: Improvement > Components: native >Affects Versions: 3.0.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Major > Attachments: HDFS-14015.001.patch, HDFS-14015.002.patch, > HDFS-14015.003.patch, HDFS-14015.004.patch, HDFS-14015.005.patch, > HDFS-14015.006.patch, HDFS-14015.007.patch, HDFS-14015.008.patch > > > In the hdfsThreadDestructor() function, we ignore the return value from the > DetachCurrentThread() call. We are seeing cases where a native thread dies > while holding a JVM monitor, and it doesn't release the monitor. We're > hoping that logging this error instead of ignoring it will shed some light on > the issue. In any case, it's good programming practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-14035:
--
Attachment: HDFS-14035-HDFS-12943.005.patch
> NN status discovery does not leverage delegation token
> --
>
> Key: HDFS-14035
> URL: https://issues.apache.org/jira/browse/HDFS-14035
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-14035-HDFS-12943.001.patch,
> HDFS-14035-HDFS-12943.002.patch, HDFS-14035-HDFS-12943.003.patch,
> HDFS-14035-HDFS-12943.004.patch, HDFS-14035-HDFS-12943.005.patch
>
>
> Currently ObserverReadProxyProvider uses
> {{HAServiceProtocol#getServiceStatus}} to get the status of each NN. However
> {{HAServiceProtocol}} does not leverage delegation token. So when running an
> application on YARN and when YARN node manager makes this call
> getServiceStatus, token authentication will fail, causing the application to
> fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[
https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673837#comment-16673837
]
Hadoop QA commented on HDFS-14015:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 39s{color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client generated 9 new + 2
unchanged - 0 fixed = 11 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 56s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed CTEST tests | test_test_libhdfs_threaded_hdfs_static |
| | test_libhdfs_threaded_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14015 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946750/HDFS-14015.008.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux d5b9d1478a7a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 989715e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| cc |
https://builds.apache.org/job/PreCommit-HDFS-Build/25425/artifact/out/diff-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| CTEST |
https://builds.apache.org/job/PreCommit-HDFS-Build/25425/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25425/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25425/testReport/ |
| Max. process+thread count | 317 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25425/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error handling in hdfsThreadDestructor in native thread local storage
>
[jira] [Commented] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673852#comment-16673852
]
Íñigo Goiri commented on HDFS-14049:
Thanks [~surmountian] for the comment; attached [^HDFS-14049.001.patch] which
uses the {{getWinUtilsFile()}}.
It seems to work in my machine without changing the pom.xml.
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch, HDFS-14049.001.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673863#comment-16673863
]
Xiao Liang commented on HDFS-14049:
---
[^HDFS-14049.001.patch] looks good to me, let's wait for the result from Yetus.
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch, HDFS-14049.001.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14049:
---
Attachment: HDFS-14049.000.patch
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14049:
---
Labels: windows (was: )
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14049:
---
Status: Patch Available (was: Open)
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-14049.000.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[ https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673741#comment-16673741 ] Daniel Templeton commented on HDFS-14015: - I'm worried about the testing, too. No idea how to reasonably test this code. I'll work on adding the extra info. > Improve error handling in hdfsThreadDestructor in native thread local storage > - > > Key: HDFS-14015 > URL: https://issues.apache.org/jira/browse/HDFS-14015 > Project: Hadoop HDFS > Issue Type: Improvement > Components: native >Affects Versions: 3.0.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Major > Attachments: HDFS-14015.001.patch, HDFS-14015.002.patch, > HDFS-14015.003.patch, HDFS-14015.004.patch, HDFS-14015.005.patch, > HDFS-14015.006.patch, HDFS-14015.007.patch > > > In the hdfsThreadDestructor() function, we ignore the return value from the > DetachCurrentThread() call. We are seeing cases where a native thread dies > while holding a JVM monitor, and it doesn't release the monitor. We're > hoping that logging this error instead of ignoring it will shed some light on > the issue. In any case, it's good programming practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673740#comment-16673740
]
Íñigo Goiri commented on HDFS-13834:
I was just thinking on doing:
{code}
GenericTestUtils.waitFor(()->queue.isEmpty(), 50, 5000);
{code}
In this way we wait at most 5 seconds or so.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673749#comment-16673749
]
CR Hota commented on HDFS-13834:
[~elgoiri] Yeah, this is what I was mentioning will get into tricky area. For
ex after 5 secs, we cannot simply assume that thread has picked up the task and
worked, we would again check the queue and assert if task is still present,
(its possible that even after 5sec task is still present), if task is present
then we cant assert that thread is alive, coz the thread will surely be alive
as it hasn't done any work and we cant assert if it will be fine if it were to
pick the task and work on it.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673783#comment-16673783
]
Hadoop QA commented on HDFS-14035:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
1s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
53s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
24s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
6s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
37s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
6s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 6s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 6s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 42s{color} | {color:orange} root: The patch generated 24 new + 290 unchanged
- 1 fixed = 314 total (was 291) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 47s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
30s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}237m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.protocol.TestReadOnly |
| | hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestConsistentReadsObserver |
| | hadoop.hdfs.server.namenode.ha.TestMultiObserverNode |
| | hadoop.hdfs.TestStateAlignmentContextWithHA |
| |
[jira] [Commented] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673817#comment-16673817
]
Chen Liang commented on HDFS-14035:
---
v004 patch had another bug, post v005 patch to fix. Also, still exploring the
possibility of separating getServiceStatus to a new protocol.
> NN status discovery does not leverage delegation token
> --
>
> Key: HDFS-14035
> URL: https://issues.apache.org/jira/browse/HDFS-14035
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-14035-HDFS-12943.001.patch,
> HDFS-14035-HDFS-12943.002.patch, HDFS-14035-HDFS-12943.003.patch,
> HDFS-14035-HDFS-12943.004.patch, HDFS-14035-HDFS-12943.005.patch
>
>
> Currently ObserverReadProxyProvider uses
> {{HAServiceProtocol#getServiceStatus}} to get the status of each NN. However
> {{HAServiceProtocol}} does not leverage delegation token. So when running an
> application on YARN and when YARN node manager makes this call
> getServiceStatus, token authentication will fail, causing the application to
> fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673825#comment-16673825
]
Xiao Liang commented on HDFS-14049:
---
Thanks [~elgoiri] for the fix, you could use getWinUtilsFile() for winutils.exe
instead.
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673886#comment-16673886
]
Hadoop QA commented on HDFS-14049:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 37m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14049 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946766/HDFS-14049.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f2a9d1abe52b 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 989715e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25428/testReport/ |
| Max. process+thread count | 667 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25428/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
>
[jira] [Commented] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673805#comment-16673805
]
Hadoop QA commented on HDFS-14049:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
21s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14049 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946741/HDFS-14049.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 48594a5cabc1 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aed836e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25424/testReport/ |
| Max. process+thread count | 658 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25424/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Updated] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-14035:
--
Attachment: HDFS-14035-HDFS-12943.006.patch
> NN status discovery does not leverage delegation token
> --
>
> Key: HDFS-14035
> URL: https://issues.apache.org/jira/browse/HDFS-14035
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-14035-HDFS-12943.001.patch,
> HDFS-14035-HDFS-12943.002.patch, HDFS-14035-HDFS-12943.003.patch,
> HDFS-14035-HDFS-12943.004.patch, HDFS-14035-HDFS-12943.005.patch,
> HDFS-14035-HDFS-12943.006.patch
>
>
> Currently ObserverReadProxyProvider uses
> {{HAServiceProtocol#getServiceStatus}} to get the status of each NN. However
> {{HAServiceProtocol}} does not leverage delegation token. So when running an
> application on YARN and when YARN node manager makes this call
> getServiceStatus, token authentication will fail, causing the application to
> fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673880#comment-16673880
]
Íñigo Goiri commented on HDFS-13834:
HDFS-13404 already tracks the flaky test in TestRouterWebHDFSContractAppend.
+1 on [^HDFS-13834-HDFS-13891.5.patch].
Committing to HDFS-13891.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834-HDFS-13891.5.patch, HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673933#comment-16673933
]
Hadoop QA commented on HDFS-14035:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
39s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
30s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
3s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
24s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
15s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
55s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 55s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 55s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 42s{color} | {color:orange} root: The patch generated 32 new + 189 unchanged
- 1 fixed = 221 total (was 190) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 52s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
34s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}269m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.protocol.TestReadOnly |
| | hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14035 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673873#comment-16673873
]
Hadoop QA commented on HDFS-13834:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
40s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 16s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13834 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946761/HDFS-13834-HDFS-13891.5.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1226004dc27e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 4b5fd4b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25427/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25427/testReport/ |
| Max. process+thread count | 952 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Commented] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673891#comment-16673891
]
Hadoop QA commented on HDDS-796:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} framework in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-796 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946768/HDDS-796.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 68fa0e577ef8 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 989715e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1602/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/framework U: hadoop-hdds/framework |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1602/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
[jira] [Commented] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[
https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673743#comment-16673743
]
Hadoop QA commented on HDFS-14015:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
42m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
41s{color} | {color:red} hadoop-hdfs-native-client in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 41s{color} |
{color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 41s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14015 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946734/HDFS-14015.007.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 126db8628c16 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aed836e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| cc |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/testReport/ |
| Max. process+thread count | 323 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25422/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error handling in hdfsThreadDestructor in native thread local storage
>
[jira] [Created] (HDFS-14051) Refactor NameNodeHttpServer#initWebHdfs to specify local keytab
Íñigo Goiri created HDFS-14051:
--
Summary: Refactor NameNodeHttpServer#initWebHdfs to specify local
keytab
Key: HDFS-14051
URL: https://issues.apache.org/jira/browse/HDFS-14051
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Íñigo Goiri
Assignee: CR Hota
We use {{NameNodeHttpServer#initWebHdfs}} from {{RouterHttpServer}}.
However, this relies on {{NameNodeHttpServer#getAuthFilterParams()}} which uses:
{code}
String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
{code}
We should refactor this to be able to specify the keytab file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14051) Refactor NameNodeHttpServer#initWebHdfs to specify local keytab
[
https://issues.apache.org/jira/browse/HDFS-14051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673780#comment-16673780
]
Íñigo Goiri commented on HDFS-14051:
This is needed by HDFS-12284 as we need to specify the keytab for the Router.
> Refactor NameNodeHttpServer#initWebHdfs to specify local keytab
> ---
>
> Key: HDFS-14051
> URL: https://issues.apache.org/jira/browse/HDFS-14051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
>
> We use {{NameNodeHttpServer#initWebHdfs}} from {{RouterHttpServer}}.
> However, this relies on {{NameNodeHttpServer#getAuthFilterParams()}} which
> uses:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> We should refactor this to be able to specify the keytab file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-798) Storage-class is showing incorrectly
[
https://issues.apache.org/jira/browse/HDDS-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-798:
Target Version/s: 0.3.0, 0.4.0
> Storage-class is showing incorrectly
>
>
> Key: HDDS-798
> URL: https://issues.apache.org/jira/browse/HDDS-798
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>
> After HDDS-712, we support storage-class.
> For list-objects, even if key has set storage-class to REDUCED_REDUNDANCY,
> still it shows STANDARD.
> As in code in list object response, we have hardcoded it as below.
> keyMetadata.setStorageClass("STANDARD");
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14049) TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
[
https://issues.apache.org/jira/browse/HDFS-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14049:
---
Attachment: HDFS-14049.001.patch
> TestHttpFSServerWebServer fails on Windows because of missing winutils.exe
> --
>
> Key: HDFS-14049
> URL: https://issues.apache.org/jira/browse/HDFS-14049
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: windows
> Attachments: HDFS-14049.000.patch, HDFS-14049.001.patch
>
>
> 2018-11-02 15:00:51,780 WARN Shell - Did not find winutils.exe: {}
> java.io.FileNotFoundException: Could not locate Hadoop executable:
> D:\hadoop-2.9\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\test-dir\bin\winutils.exe
> -see https://wiki.apache.org/hadoop/WindowsProblems
> at org.apache.hadoop.util.Shell.getQualifiedBinInner(Shell.java:612)
> at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:585)
> at org.apache.hadoop.util.Shell.(Shell.java:682)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
> at
> org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1587)
> at
> org.apache.hadoop.fs.http.server.HttpFSServerWebServer.(HttpFSServerWebServer.java:93)
> at
> org.apache.hadoop.fs.http.server.TestHttpFSServerWebServer.setUp(TestHttpFSServerWebServer.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14035) NN status discovery does not leverage delegation token
[
https://issues.apache.org/jira/browse/HDFS-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673877#comment-16673877
]
Hadoop QA commented on HDFS-14035:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
15s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
37s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
3s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
10s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
41s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 47s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 47s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 19s{color} | {color:orange} root: The patch generated 22 new + 188 unchanged
- 1 fixed = 210 total (was 189) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
36s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
41s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
46s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 39s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-hdfs-client in the patch failed.
[jira] [Created] (HDFS-14050) Use parameterized logging construct in NamenodeFsck class
Hrishikesh Gadre created HDFS-14050: --- Summary: Use parameterized logging construct in NamenodeFsck class Key: HDFS-14050 URL: https://issues.apache.org/jira/browse/HDFS-14050 Project: Hadoop HDFS Issue Type: Task Affects Versions: 3.0.0 Reporter: Hrishikesh Gadre Assignee: Hrishikesh Gadre HDFS-13695 implemented a change to use slf4j logger (instead of commons logging). But the NamenodeFsck class is still not using parameterized logging construct. This came up during the code review for HADOOP-11391. We should change logging statements in NamenodeFsck to use slf4j style parameterized logging apis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-797) If DN is started before SCM, it does not register
[ https://issues.apache.org/jira/browse/HDDS-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673773#comment-16673773 ] Hanisha Koneru commented on HDDS-797: - This issue happens because the DN queues multiple GetVersion tasks while waiting to connect with SCM. When SCM eventually comes up, all the GetVersion tasks still in the queue (which have not timed out retrying) execute successfully and update the endpoint state of the DN state machine. The first GetVersion task to successfully get the version from SCM updates the state from GETVERSION to HEARTBEAT. The second GetVersion task then updates this state from HEARTBEAT to SHUTDOWN. Once DN goes into HEARTBEAT state, it should not execute the GetVersion tasks. > If DN is started before SCM, it does not register > - > > Key: HDDS-797 > URL: https://issues.apache.org/jira/browse/HDDS-797 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: Hanisha Koneru >Priority: Blocker > Attachments: HDDS-797.001.patch > > > If a DN is started before SCM, it does not register with the SCM. DNs keep > trying to connect with the SCM and once SCM is up, the DN services are > shutdown instead of registering with SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673771#comment-16673771
]
Íñigo Goiri commented on HDFS-13834:
I agree on leaving the check for alive and so on.
I just want to replace the {{while}} that can last forever with an the waitFor
limited to 5 seconds (or whatever value is reasonable).
It would be the same behavior as what [^HDFS-13834-HDFS-13891.4.patch] is doing
right now.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14015) Improve error handling in hdfsThreadDestructor in native thread local storage
[
https://issues.apache.org/jira/browse/HDFS-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673790#comment-16673790
]
Daniel Templeton commented on HDFS-14015:
-
New patch uses {{toString() + ":" + getId()}} and removes some of the obsessive
and unnecessary error checking.
> Improve error handling in hdfsThreadDestructor in native thread local storage
> -
>
> Key: HDFS-14015
> URL: https://issues.apache.org/jira/browse/HDFS-14015
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: native
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HDFS-14015.001.patch, HDFS-14015.002.patch,
> HDFS-14015.003.patch, HDFS-14015.004.patch, HDFS-14015.005.patch,
> HDFS-14015.006.patch, HDFS-14015.007.patch, HDFS-14015.008.patch
>
>
> In the hdfsThreadDestructor() function, we ignore the return value from the
> DetachCurrentThread() call. We are seeing cases where a native thread dies
> while holding a JVM monitor, and it doesn't release the monitor. We're
> hoping that logging this error instead of ignoring it will shed some light on
> the issue. In any case, it's good programming practice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-13834:
---
Attachment: HDFS-13834-HDFS-13891.5.patch
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834-HDFS-13891.4.patch,
> HDFS-13834-HDFS-13891.5.patch, HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673867#comment-16673867
]
Yiqun Lin commented on HDDS-796:
Thanks for the review, [~giovanni.fumarola].
Attach the updated patch.
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
>
> Key: HDDS-796
> URL: https://issues.apache.org/jira/browse/HDDS-796
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-796.001.patch, HDDS-796.002.patch
>
>
> After replacing usage of Guava Optional with Java Optional (HDDS-751), there
> are two unit test failures generated
> ([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
> unit tests are not fully triggered and tested across all the projects under
> HDDS-751, :P
> {noformat}
> java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
> at
> org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDDS-796:
---
Attachment: HDDS-796.002.patch
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
>
> Key: HDDS-796
> URL: https://issues.apache.org/jira/browse/HDDS-796
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-796.001.patch, HDDS-796.002.patch
>
>
> After replacing usage of Guava Optional with Java Optional (HDDS-751), there
> are two unit test failures generated
> ([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
> unit tests are not fully triggered and tested across all the projects under
> HDDS-751, :P
> {noformat}
> java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
> at
> org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-766) Ozone shell create volume fails if volume name does not have a leading slash
[
https://issues.apache.org/jira/browse/HDDS-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672658#comment-16672658
]
chencan commented on HDDS-766:
--
testCreateVolumeWithoutScheme run fail in TestOzoneShell.
Test cases in TestOzoneShell will add complete host:port in url, but "ozone sh
volume create volume1"
take RpcClient as default clientProtocol without host:port，
we should set "ozone.om.address" in conf before test. I'm not sure if this is
the right way.
> Ozone shell create volume fails if volume name does not have a leading slash
>
>
> Key: HDDS-766
> URL: https://issues.apache.org/jira/browse/HDDS-766
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: chencan
>Priority: Major
> Labels: newbie
> Attachments: HDDS-766.001.patch, HDDS-766.003.patch
>
>
> After HDDS-682, volume creation through shell fails if the volume name does
> not have leading slash.
> {code:java}
> $ ./ozone sh volume create volume1
> Volume name is required
> $ ./ozone sh volume create /volume1
> 2018-10-30 14:07:58,078 INFO rpc.RpcClient: Creating Volume: volume1, with
> hdds as owner and quota set to 1152921504606846976 bytes.{code}
> In {{OzoneAddress#stringToUri}}, when creating a new URI, the path parameter
> is expected to have a leading slash. Otherwise, the path gets mixed with the
> authority.
> To fix this, we should add a leading slash to the path variable, if it does
> not exist, before constructing the URI object.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14045) Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN
[ https://issues.apache.org/jira/browse/HDFS-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672683#comment-16672683 ] Jiandan Yang commented on HDFS-14045: -- [~cheersyang] Would you please help me review this patch? > Use different metrics in DataNode to better measure latency of > heartbeat/blockReports/incrementalBlockReports of Active/Standby NN > -- > > Key: HDFS-14045 > URL: https://issues.apache.org/jira/browse/HDFS-14045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: HDFS-14045.001.patch > > > Currently DataNode uses same metrics to measure rpc latency of NameNode, but > Active and Standby usually have different performance at the same time, > especially in large cluster. For example, rpc latency of Standby is very long > when Standby is catching up editlog. We may misunderstand the state of HDFS. > Using different metrics for Active and standby can help us obtain more > precise metric data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13984) getFileInfo of libhdfs call NameNode#getFileStatus twice
[
https://issues.apache.org/jira/browse/HDFS-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672654#comment-16672654
]
Jiandan Yang commented on HDFS-13984:
--
[~jzhuge] Could you help me review this patch?
> getFileInfo of libhdfs call NameNode#getFileStatus twice
>
>
> Key: HDFS-13984
> URL: https://issues.apache.org/jira/browse/HDFS-13984
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: HDFS-13984.001.patch, HDFS-13984.002.patch
>
>
> getFileInfo in hdfs.c calls *FileSystem#exists* first, then calls
> *FileSystem#getFileStatus*.
> *FileSystem#exists* also call *FileSystem#getFileStatus*, just as follows:
> {code:java}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> and finally this leads to call NameNodeRpcServer#getFileInfo twice.
> Actually we can implement by calling once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[ https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672663#comment-16672663 ] Zsolt Horvath commented on HDDS-692: I'm checking the patch, as locally it succeeded. > Use the ProgressBar class in the RandomKeyGenerator freon test > -- > > Key: HDDS-692 > URL: https://issues.apache.org/jira/browse/HDDS-692 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Reporter: Elek, Marton >Assignee: Zsolt Horvath >Priority: Major > Attachments: HDDS-692.001.patch, HDDS-692.002.patch > > > HDDS-443 provides a reusable progress bar to make it easier to add more freon > tests, but the existing RandomKeyGenerator test > (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java) > still doesn't use it. > It would be good to switch to use the new progress bar there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[ https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Horvath updated HDDS-692: --- Status: Open (was: Patch Available) > Use the ProgressBar class in the RandomKeyGenerator freon test > -- > > Key: HDDS-692 > URL: https://issues.apache.org/jira/browse/HDDS-692 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Reporter: Elek, Marton >Assignee: Zsolt Horvath >Priority: Major > Attachments: HDDS-692.001.patch, HDDS-692.002.patch > > > HDDS-443 provides a reusable progress bar to make it easier to add more freon > tests, but the existing RandomKeyGenerator test > (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java) > still doesn't use it. > It would be good to switch to use the new progress bar there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-790) Support multipart upload on Ozone S3 gateway interface
Elek, Marton created HDDS-790: - Summary: Support multipart upload on Ozone S3 gateway interface Key: HDDS-790 URL: https://issues.apache.org/jira/browse/HDDS-790 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: S3 Reporter: Elek, Marton AWS S3 Muptipart uploads API enables to upload huge files in chunks. Documented here: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html Should be investigated how is it possible with OzoneClient. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-793) Support custom key/value annotations on volume/bucket/key
Elek, Marton created HDDS-793: - Summary: Support custom key/value annotations on volume/bucket/key Key: HDDS-793 URL: https://issues.apache.org/jira/browse/HDDS-793 Project: Hadoop Distributed Data Store Issue Type: New Feature Components: OM Reporter: Elek, Marton Assignee: Elek, Marton I propose to add a custom Map annotation field to objects/buckets and keys in Ozone Manager. It would enable to build any extended functionality on top of the OM's generic interface. For example: * Support tags in Ozone S3 gateway (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGETtagging.html) * Support md5 based ETags in s3g * Store s3 related authorization data (ACLs, policies) together with the parent objects As an optional feature (could be implemented later) the client can defined the exposed annotations. For example s3g can defined which annotations should be read from rocksdb on OM side and sent the the client (s3g) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-795) RocksDb specific classes leak from DBStore/Table interfaces
Elek, Marton created HDDS-795:
-
Summary: RocksDb specific classes leak from DBStore/Table
interfaces
Key: HDDS-795
URL: https://issues.apache.org/jira/browse/HDDS-795
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Elek, Marton
Assignee: Elek, Marton
org.apache.hadoop.utils.db.RocksDB and Table interfaces provide a
vendor-independent way to access any key value store.
The default implementation uses RocksDb but other implementation also could be
used (for example an InMemory implementation for testing only).
The current Table interface contains methods which depend on RocksDB specific
classes. For example:
{code}
public interface DBStore extends AutoCloseable {
//...
/**
* Return the Column Family handle. TODO: This leaks an RockDB abstraction
* into Ozone code, cleanup later.
*
* @return ColumnFamilyHandle
*/
ColumnFamilyHandle getHandle();
//...
{code}
We need to remove the RocksDB specific classes from the generic interfaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects
Elek, Marton created HDDS-792: - Summary: Use md5 hash as ETag for Ozone S3 objects Key: HDDS-792 URL: https://issues.apache.org/jira/browse/HDDS-792 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: S3 Reporter: Elek, Marton AWS S3 uses md5 hash of the files as ETag. Not a strict requirement, but s3 tests (https://github.com/gaul/s3-tests/) can not been executed without that. It requires to support custom key/value annotations on key objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14046) There is no ICON for Maintenance In Datanode UI page and after Datanode moved into Maintenance states still datanode mark status is empty in Datanode UI.
Harshakiran Reddy created HDFS-14046: Summary: There is no ICON for Maintenance In Datanode UI page and after Datanode moved into Maintenance states still datanode mark status is empty in Datanode UI. Key: HDFS-14046 URL: https://issues.apache.org/jira/browse/HDFS-14046 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.1.1 Reporter: Harshakiran Reddy -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine
Shashikant Banerjee created HDDS-794: Summary: Add configs to set StateMachineData write timeout in ContainerStateMachine Key: HDDS-794 URL: https://issues.apache.org/jira/browse/HDDS-794 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Datanode Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.3.0, 0.4.0 The patch will address adding config settings in Ozone which will enable/disable timeout for StateMachineData write via Ratis. It also adds some debug logs in writeChunk handling path inside datanode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-794: - Attachment: HDDS-794.000.patch > Add configs to set StateMachineData write timeout in ContainerStateMachine > -- > > Key: HDDS-794 > URL: https://issues.apache.org/jira/browse/HDDS-794 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.3.0, 0.4.0 > > Attachments: HDDS-794.000.patch > > > The patch will address adding config settings in Ozone which will > enable/disable timeout for StateMachineData write via Ratis. It also adds > some debug logs in writeChunk handling path inside datanode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-795) RocksDb specific classes leak from DBStore/Table interfaces
[
https://issues.apache.org/jira/browse/HDDS-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-795:
--
Attachment: HDDS-795.001.patch
> RocksDb specific classes leak from DBStore/Table interfaces
> ---
>
> Key: HDDS-795
> URL: https://issues.apache.org/jira/browse/HDDS-795
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-795.001.patch
>
>
> org.apache.hadoop.utils.db.RocksDB and Table interfaces provide a
> vendor-independent way to access any key value store.
> The default implementation uses RocksDb but other implementation also could
> be used (for example an InMemory implementation for testing only).
> The current Table interface contains methods which depend on RocksDB specific
> classes. For example:
> {code}
> public interface DBStore extends AutoCloseable {
> //...
> /**
>* Return the Column Family handle. TODO: This leaks an RockDB abstraction
>* into Ozone code, cleanup later.
>*
>* @return ColumnFamilyHandle
>*/
> ColumnFamilyHandle getHandle();
> //...
> {code}
> We need to remove the RocksDB specific classes from the generic interfaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-642) Add chill mode exit condition for pipeline availability
[
https://issues.apache.org/jira/browse/HDDS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673069#comment-16673069
]
Hadoop QA commented on HDDS-642:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange} hadoop-hdds/server-scm: The patch generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdds/server-scm generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 22s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/server-scm |
| | Should
org.apache.hadoop.hdds.scm.server.SCMChillModeManager$PipelineChillModeRule be
a _static_ inner class? At SCMChillModeManager.java:inner class? At
SCMChillModeManager.java:[lines 305-326] |
| Failed junit tests | hadoop.hdds.scm.TestStorageContainerManagerHttpServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-642 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946658/HDDS-642.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 78467ecdff55 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d16d5f7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/1594/artifact/out/diff-checkstyle-hadoop-hdds_server-scm.txt
|
| findbugs |
[jira] [Commented] (HDFS-14024) RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService
[
https://issues.apache.org/jira/browse/HDFS-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672824#comment-16672824
]
Brahma Reddy Battula commented on HDFS-14024:
-
Reabsed now.Please have look.
> RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService
> -
>
> Key: HDFS-14024
> URL: https://issues.apache.org/jira/browse/HDFS-14024
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-14024-HDFS-13891.0.patch, HDFS-14024.0.patch
>
>
> Routers may be proxying for a downstream name node that is NOT migrated to
> understand "ProvidedCapacityTotal". updateJMXParameters method in
> NamenodeHeartbeatService should handle this without breaking.
>
> {code:java}
> jsonObject.getLong("MissingBlocks"),
> jsonObject.getLong("PendingReplicationBlocks"),
> jsonObject.getLong("UnderReplicatedBlocks"),
> jsonObject.getLong("PendingDeletionBlocks"),
> jsonObject.getLong("ProvidedCapacityTotal"));
> {code}
> One way to do this is create a json wrapper while gives back some default if
> json node is not found.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-642) Add chill mode exit condition for pipeline availability
[ https://issues.apache.org/jira/browse/HDDS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672858#comment-16672858 ] Yiqun Lin commented on HDDS-642: Thanks for the review, [~ajayydv]. I simplified the pipeline ruie check and used a atomic type. Please have a look. > Add chill mode exit condition for pipeline availability > --- > > Key: HDDS-642 > URL: https://issues.apache.org/jira/browse/HDDS-642 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Arpit Agarwal >Assignee: Yiqun Lin >Priority: Major > Attachments: HDDS-642.001.patch, HDDS-642.002.patch > > > SCM should not exit chill-mode until at least 1 write pipeline is available. > Else smoke tests are unreliable. > This is not an issue for real clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-791) Support Range header for Oozne s3 object download
Elek, Marton created HDDS-791:
-
Summary: Support Range header for Oozne s3 object download
Key: HDDS-791
URL: https://issues.apache.org/jira/browse/HDDS-791
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Components: S3
Reporter: Elek, Marton
Using s3 rest api smaller chunks of an object could be uploaded with using
Range headers:
For example:
{code}
GET /example-object HTTP/1.1
Host: example-bucket.s3.amazonaws.com
x-amz-date: Fri, 28 Jan 2011 21:32:02 GMT
Range: bytes=0-9
Authorization: AWS AKIAIOSFODNN7EXAMPLE:Yxg83MZaEgh3OZ3l0rLo5RTX11o=
Sample Response with Specified Range of the Object Bytes
{code}
Can be implemented with using the seek method on OzoneInputStream.
The Range header support is one of the missing piece for fully support s3a
interface.
References:
Range header spec:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
Aws s3 doc:
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14046) There is no ICON for Maintenance In Datanode UI page and after Datanode moved into Maintenance states still datanode mark status is empty in Datanode UI.
[ https://issues.apache.org/jira/browse/HDFS-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar reassigned HDFS-14046: Assignee: Ranith Sardar > There is no ICON for Maintenance In Datanode UI page and after Datanode moved > into Maintenance states still datanode mark status is empty in Datanode UI. > -- > > Key: HDFS-14046 > URL: https://issues.apache.org/jira/browse/HDFS-14046 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Ranith Sardar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-794: - Status: Patch Available (was: Open) > Add configs to set StateMachineData write timeout in ContainerStateMachine > -- > > Key: HDDS-794 > URL: https://issues.apache.org/jira/browse/HDDS-794 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.3.0, 0.4.0 > > Attachments: HDDS-794.000.patch > > > The patch will address adding config settings in Ozone which will > enable/disable timeout for StateMachineData write via Ratis. It also adds > some debug logs in writeChunk handling path inside datanode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-642) Add chill mode exit condition for pipeline availability
[ https://issues.apache.org/jira/browse/HDDS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDDS-642: --- Attachment: HDDS-642.002.patch > Add chill mode exit condition for pipeline availability > --- > > Key: HDDS-642 > URL: https://issues.apache.org/jira/browse/HDDS-642 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Arpit Agarwal >Assignee: Yiqun Lin >Priority: Major > Attachments: HDDS-642.001.patch, HDDS-642.002.patch > > > SCM should not exit chill-mode until at least 1 write pipeline is available. > Else smoke tests are unreliable. > This is not an issue for real clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14045) Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN
[ https://issues.apache.org/jira/browse/HDFS-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672845#comment-16672845 ] Weiwei Yang commented on HDFS-14045: Hi [~yangjiandan] Thanks for creating this and working on this. Some high-level questions regarding to the patch, # does this work in non-HA environment? # does this work if HDFS federation is enabled? I am thinking, maybe we can use namespaceID (as that is part of the DatanodeRegistration info) to differentiate metrics for different namenodes? Would that be more general? Not sure if the metrics system allow us to dynamically add new entries though. Thanks > Use different metrics in DataNode to better measure latency of > heartbeat/blockReports/incrementalBlockReports of Active/Standby NN > -- > > Key: HDFS-14045 > URL: https://issues.apache.org/jira/browse/HDFS-14045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: HDFS-14045.001.patch > > > Currently DataNode uses same metrics to measure rpc latency of NameNode, but > Active and Standby usually have different performance at the same time, > especially in large cluster. For example, rpc latency of Standby is very long > when Standby is catching up editlog. We may misunderstand the state of HDFS. > Using different metrics for Active and standby can help us obtain more > precise metric data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-748) Use strongly typed metadata Table implementation
[
https://issues.apache.org/jira/browse/HDDS-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-748:
--
Issue Type: Improvement (was: Wish)
Summary: Use strongly typed metadata Table implementation (was: Use
strongly typed metadata Table implementation (proposal))
Thanks [~anu] the feedback. Removed the proposal tag.
I also added a link: I think it blocks HDDS-763 (that would be more easy with
separated codecs.
> Use strongly typed metadata Table implementation
>
>
> Key: HDDS-748
> URL: https://issues.apache.org/jira/browse/HDDS-748
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> NOTE: This issue is a proposal. I assigned it to myself to make it clear that
> it's not ready to implement, I just start a discussion about the proposed
> change.
> org.apache.hadoop.utils.db.DBStore (from HDDS-356) is a new generation
> MetadataStore to store all persistent state of hdds/ozone scm/om/datanodes.
> It supports column families with via the Table interface which supports
> methods like:
> {code:java}
> byte[] get(byte[] key) throws IOException;
> void put(byte[] key, byte[] value)
> {code}
> In our current code we usually use static helpers to do the _byte[] ->
> object_ and _object -> byte[]_ conversion with protobuf.
> For example in KeyManagerImpl the OmKeyInfo.getFromProtobuf is used multiple
> times to deserialize the OmKeyInfo project.
>
> *I propose to create a type-safe table* with using:
> {code:java}
> public interface Table extends AutoCloseable
> {code}
> The put and get could be modified to:
> {code:java}
> VALUE_TYPE[] get(KEY_TYPE[] key) throws IOException;
> void put(KEY_TYPE[] key, VALUE_TYPE value)
> {code}
> For example for the key table it could be:
> {code:java}
> OmKeyInfo get(String key) throws IOException;
> void put(String key, OmKeyInfo value)
> {code}
>
> It requires to register internal codec (marshaller/unmarshaller)
> implementations during the creation of (..)Table.
> The registration of the codecs would be optional. Without it the Table could
> work as now (using byte[],byte[])
> *Advantages*:
> * More simplified code (Don't need to repeat the serialization everywhere)
> less error-prone.
> * Clear separation of the layers (As of now I can't see the serialization
> overhead with OpenTracing) and measurablity). Easy to test different
> serialization in the future.
> * Easier to create additional developer tools to investigate the current
> state of the rocksdb metadata stores. We had SQLCLI to export all the data to
> sql, but with registering the format in the rocksdb table we can easily
> create a calcite based SQL console.
> *Additional info*:
> I would modify the interface of the DBStoreBuilder and DBStore:
> {code:java}
>this.store = DBStoreBuilder.newBuilder(conf)
> .setName(OM_DB_NAME)
> .setPath(Paths.get(metaDir.getPath()))
> .addTable(KEY_TABLE, DBUtil.STRING_KEY_CODEC, new OmKeyInfoCoder())
> //...
> .build();
> {code}
> And using it from the DBStore:
> {code:java}
> //default, without codec
> Table getTable(String name) throws IOException;
> //advanced with codec from the codec registry
> Table getTable(String name, Class keyType, Class valueType);
> //for example
> table.getTable(KEY_TABLE,String.class,OmKeyInfo.class);
> //or
> table.getTable(KEY_TABLE,String.class,UserInfo.class)
> //exception is thrown: No codec is registered for KEY_TABLE with type
> UserInfo.{code}
> *Priority*:
> I think it's a very useful and valuable step forward but the real priority is
> lower. Ideal for new contributors especially as it's independent, standalone
> part of ozone code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-795) RocksDb specific classes leak from DBStore/Table interfaces
[
https://issues.apache.org/jira/browse/HDDS-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-795:
--
Status: Patch Available (was: Open)
Patch is uploaded:
No change in the logic just OOP level reorganization.
DBStore can create/commit batch operation (RocksDB independent interface)
{code}
public interface DBStore extends AutoCloseable {
/**
* A init a holder for PUT, DELETE operations handled as a single atomic
* write.
*
* @throws IOException BatchOperation initialization fails.
*/
BatchOperation initBatchOperation();
/**
* Commit the batch operations .
*
* @param operation to de
* @throws IOException write fails
*/
void commitBatchOperation(BatchOperation operation) throws IOException;
}
{code}
And Table can put/delete immediately or with the help of a batch operation:
{code}
void put(byte[] key, byte[] value) throws IOException;
void put( batch, byte[] key, byte[] value) throws
IOException;
{code}
> RocksDb specific classes leak from DBStore/Table interfaces
> ---
>
> Key: HDDS-795
> URL: https://issues.apache.org/jira/browse/HDDS-795
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-795.001.patch
>
>
> org.apache.hadoop.utils.db.RocksDB and Table interfaces provide a
> vendor-independent way to access any key value store.
> The default implementation uses RocksDb but other implementation also could
> be used (for example an InMemory implementation for testing only).
> The current Table interface contains methods which depend on RocksDB specific
> classes. For example:
> {code}
> public interface DBStore extends AutoCloseable {
> //...
> /**
>* Return the Column Family handle. TODO: This leaks an RockDB abstraction
>* into Ozone code, cleanup later.
>*
>* @return ColumnFamilyHandle
>*/
> ColumnFamilyHandle getHandle();
> //...
> {code}
> We need to remove the RocksDB specific classes from the generic interfaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14022) Failing CTEST test_libhdfs
[
https://issues.apache.org/jira/browse/HDFS-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673095#comment-16673095
]
Anatoli Shein commented on HDFS-14022:
--
Looks like the remaining failures on trunk are caused by
hdfsGetLastExceptionRootCause function call in test_libhdfs_threaded.c. I just
created HDFS-14047 to look into this.
> Failing CTEST test_libhdfs
> --
>
> Key: HDFS-14022
> URL: https://issues.apache.org/jira/browse/HDFS-14022
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Major
>
> Here are list of recurring failures that are seen, there seems to be a
> problem with
> invoking the build() in MiniDFSClusterBuilder, there are several failures ( 2
> cores related to it), in the function
> struct NativeMiniDfsCluster* nmdCreate(struct NativeMiniDfsConf *conf)
> {
>jthr = invokeMethod(env, , INSTANCE, bld, MINIDFS_CLUSTER_BUILDER,
> "build", "()L" MINIDFS_CLUSTER ";"); --->
> }
> Failed CTEST tests
> test_test_libhdfs_threaded_hdfs_static
> test_test_libhdfs_zerocopy_hdfs_static
> test_libhdfs_threaded_hdfspp_test_shim_static
> test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
> libhdfs_mini_stress_valgrind_hdfspp_test_static
> memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
> test_libhdfs_mini_stress_hdfspp_test_shim_static
> test_hdfs_ext_hdfspp_test_shim_static
>
> Details of the failures:
> a) test_test_libhdfs_threaded_hdfs_static
> hdfsOpenFile(/tlhData0001/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException) --->
> (unable to get stack trace for java.io.FileNotFoundException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
> with NULL return return value (errno: 2): expected substring: File does not
> exist
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
> with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti,
> fs, )
> hdfsOpenFile(/tlhData/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException)
> b) test_test_libhdfs_zerocopy_hdfs_static
> nmdCreate: Builder#build error:
> (unable to get root cause for java.lang.RuntimeException)
> (unable to get stack trace for java.lang.RuntimeException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_zerocopy.c:253
> (errno: 2): got NULL from cl
> Failure:
> struct NativeMiniDfsCluster* nmdCreate(struct NativeMiniDfsConf *conf)
> jthr = invokeMethod(env, , INSTANCE, bld, MINIDFS_CLUSTER_BUILDER,
> "build", "()L" MINIDFS_CLUSTER ";"); ===> Failure
> if (jthr) {
> printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "nmdCreate: Builder#build");
> goto error;
> }
> }
> c) test_libhdfs_threaded_hdfspp_test_shim_static
> hdfsOpenFile(/tlhData0002/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException) --->
> (unable to get stack trace for java.io.FileNotFoundException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
> with NULL return return value (errno: 2): expected substring: File does not
> exist
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
> with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti,
> fs, )
> d)
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x0078c513, pid=16765, tid=0x7fc4449717c0
> #
> # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build
> 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [hdfs_ext_hdfspp_test_shim_static+0x38c513]
> #
> # Core dump written. Default location:
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/core
> or core.16765
> #
> # An error report file with more information is saved as:
> #
>
[jira] [Work started] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14047 started by Anatoli Shein. > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[ https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Horvath updated HDDS-692: --- Attachment: HDDS-692.003.patch > Use the ProgressBar class in the RandomKeyGenerator freon test > -- > > Key: HDDS-692 > URL: https://issues.apache.org/jira/browse/HDDS-692 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Reporter: Elek, Marton >Assignee: Zsolt Horvath >Priority: Major > Attachments: HDDS-692.001.patch, HDDS-692.002.patch, > HDDS-692.003.patch > > > HDDS-443 provides a reusable progress bar to make it easier to add more freon > tests, but the existing RandomKeyGenerator test > (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java) > still doesn't use it. > It would be good to switch to use the new progress bar there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDDS-796:
---
Description:
After replacing usage of Guava Optional with Java Optional (HDDS-751), there
are two unit test failures generated
([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
unit tests are not fully triggered and tested across all the projects under
HDDS-751, :P
{noformat}
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
at
org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
at
org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
{noformat}
was:
After replacing usage of Guava Optional with Java Optional (HDDS-751), there
are two unit test failures generated
([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
unit tests are not fully triggered and tested under HDDS-751, :P
{noformat}
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
at
org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
at
org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
{noformat}
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
>
> Key: HDDS-796
> URL: https://issues.apache.org/jira/browse/HDDS-796
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-796.001.patch
>
>
> After replacing usage of Guava Optional with Java Optional (HDDS-751), there
> are two unit test failures generated
> ([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
> unit tests are not fully triggered and tested across all the projects under
> HDDS-751, :P
> {noformat}
> java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
> at
> org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Attachment: HDFS-14047.000.patch > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Status: Patch Available (was: In Progress) Patch 000 - comment out hdfsGetLastExceptionRootCause first to make sure CI tests pass this way. > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[
https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673153#comment-16673153
]
Zsolt Horvath edited comment on HDDS-692 at 11/2/18 2:22 PM:
-
{quote}{{I'm trying to resubmit the patch with these local results:}}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.ozone.scm.TestContainerSQLCli
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.896 s
- in org.apache.hadoop.ozone.scm.TestContainerSQLCli
[INFO] Running
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 s
- in org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations
[INFO] Running org.apache.hadoop.ozone.om.TestOmSQLCli
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.066
s - in org.apache.hadoop.ozone.om.TestOmSQLCli
[INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeRestart
[INFO] Running org.apache.hadoop.ozone.freon.TestProgressBar
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.518 s
- in org.apache.hadoop.ozone.freon.TestProgressBar
[INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.368 s
- in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator
[INFO] Running org.apache.hadoop.ozone.freon.TestDataValidate
[INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeFastRestart
{quote}
was (Author: horzsolt2006):
{{I'm trying to resubmit the patch with these local results:}}
{{INFO] ---}}
{{ [INFO] T E S T S}}
{{ [INFO] ---}}
{{ [INFO] Running org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
14.896 s - in org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Running
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.402 s - in
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Running org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.066 s - in org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeRestart}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.518 s - in org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.368 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestDataValidate}}
{{ [INFO] Running
org.apache.hadoop.ozone.freon.TestFreonWithDatanodeFastRestart}}
> Use the ProgressBar class in the RandomKeyGenerator freon test
> --
>
> Key: HDDS-692
> URL: https://issues.apache.org/jira/browse/HDDS-692
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Tools
>Reporter: Elek, Marton
>Assignee: Zsolt Horvath
>Priority: Major
> Attachments: HDDS-692.001.patch, HDDS-692.002.patch,
> HDDS-692.003.patch
>
>
> HDDS-443 provides a reusable progress bar to make it easier to add more freon
> tests, but the existing RandomKeyGenerator test
> (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java)
> still doesn't use it.
> It would be good to switch to use the new progress bar there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14048:
---
Attachment: HDFS-14048.000.patch
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
Anatoli Shein created HDFS-14047: Summary: [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c Key: HDFS-14047 URL: https://issues.apache.org/jira/browse/HDFS-14047 Project: Hadoop HDFS Issue Type: Sub-task Components: libhdfs, native Reporter: Anatoli Shein Assignee: Anatoli Shein Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
Erik Krogen created HDFS-14048:
--
Summary: DFSOutputStream close() throws exception on subsequent
call after DataNode restart
Key: HDFS-14048
URL: https://issues.apache.org/jira/browse/HDFS-14048
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.3.0
Reporter: Erik Krogen
Assignee: Erik Krogen
We recently discovered an issue in which, during a rolling upgrade, some jobs
were failing with exceptions like (sadly this is the whole stack trace):
{code}
java.io.IOException: A datanode is restarting:
DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
{code}
with an earlier statement in the log like:
{code}
INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
{code}
Strangely we did not see any other logs about the {{DFSOutputStream}} failing
after waiting for the DataNode restart. We eventually realized that in some
cases {{DFSOutputStream#close()}} may be called more than once, and that if so,
the {{IOException}} above is thrown on the _second_ call to {{close()}} (this
is even with HDFS-5335; prior to this it would have been thrown on all calls to
{{close()}} besides the first).
The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
new output stream is created, it resets the error states:
{code}
errorState.resetInternalError();
// remove all restarting nodes from failed nodes list
failed.removeAll(restartingNodes);
restartingNodes.clear();
{code}
But it forgets to clear {{lastException}}. When {{DFSOutputStream#closeImpl()}}
is called a second time, this block is triggered:
{code}
if (isClosed()) {
LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
closed, getStreamer().streamerClosed());
try {
getStreamer().getLastException().check(true);
{code}
The second time, {{isClosed()}} is true, so the exception checking occurs and
the "Datanode is restarting" exception is thrown even though the stream has
already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-14048 started by Erik Krogen.
--
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-642) Add chill mode exit condition for pipeline availability
[ https://issues.apache.org/jira/browse/HDDS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673174#comment-16673174 ] Yiqun Lin commented on HDDS-642: Failed unit test is not related. Attach the updated patch to fix checkstyle and findbugs issues. > Add chill mode exit condition for pipeline availability > --- > > Key: HDDS-642 > URL: https://issues.apache.org/jira/browse/HDDS-642 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Arpit Agarwal >Assignee: Yiqun Lin >Priority: Major > Attachments: HDDS-642.001.patch, HDDS-642.002.patch, > HDDS-642.003.patch > > > SCM should not exit chill-mode until at least 1 write pipeline is available. > Else smoke tests are unreliable. > This is not an issue for real clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-642) Add chill mode exit condition for pipeline availability
[ https://issues.apache.org/jira/browse/HDDS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDDS-642: --- Attachment: HDDS-642.003.patch > Add chill mode exit condition for pipeline availability > --- > > Key: HDDS-642 > URL: https://issues.apache.org/jira/browse/HDDS-642 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Arpit Agarwal >Assignee: Yiqun Lin >Priority: Major > Attachments: HDDS-642.001.patch, HDDS-642.002.patch, > HDDS-642.003.patch > > > SCM should not exit chill-mode until at least 1 write pipeline is available. > Else smoke tests are unreliable. > This is not an issue for real clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-788) Change title page of bucket browser in S3gateway
[ https://issues.apache.org/jira/browse/HDDS-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-788: Resolution: Fixed Fix Version/s: 0.4.0 0.3.0 Status: Resolved (was: Patch Available) Thank You [~ajayydv] for review. I have committed to trunk and ozone-0.3 > Change title page of bucket browser in S3gateway > > > Key: HDDS-788 > URL: https://issues.apache.org/jira/browse/HDDS-788 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 0.3.0, 0.4.0 > > Attachments: HDDS-788.00.patch, HDDS-788.01.patch > > > We need to change the name of bucket browser to Ozone S3 Explorer. > And also change title bar image to ozone icon. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[
https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673263#comment-16673263
]
Hadoop QA commented on HDFS-14047:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
29s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14047 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946684/HDFS-14047.000.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux d2a35c19cac0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d16d5f7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25417/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25417/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
>
>
> Key: HDFS-14047
> URL: https://issues.apache.org/jira/browse/HDFS-14047
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: libhdfs, native
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
>Priority: Major
> Attachments: HDFS-14047.000.patch
>
>
> Currently the native client CI tests break deterministically with these
> errors:
> Libhdfs
> 1 - test_test_libhdfs_threaded_hdfs_static (Failed)
> [exec] TEST_ERROR: failed on
>
[jira] [Commented] (HDDS-794) Add configs to set StateMachineData write timeout in ContainerStateMachine
[
https://issues.apache.org/jira/browse/HDDS-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673118#comment-16673118
]
Hadoop QA commented on HDDS-794:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdds: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-794 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946668/HDDS-794.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux b2a0378e0938 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[
https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673153#comment-16673153
]
Zsolt Horvath commented on HDDS-692:
{{I'm trying to resubmit the patch with these local results:}}
{{INFO] ---}}
{{ [INFO] T E S T S}}
{{ [INFO] ---}}
{{ [INFO] Running org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
14.896 s - in org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Running
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402
s - in org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Running org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.066 s - in org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeRestart}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.518
s - in org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.368 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestDataValidate}}
{{ [INFO] Running
org.apache.hadoop.ozone.freon.TestFreonWithDatanodeFastRestart}}
> Use the ProgressBar class in the RandomKeyGenerator freon test
> --
>
> Key: HDDS-692
> URL: https://issues.apache.org/jira/browse/HDDS-692
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Tools
>Reporter: Elek, Marton
>Assignee: Zsolt Horvath
>Priority: Major
> Attachments: HDDS-692.001.patch, HDDS-692.002.patch,
> HDDS-692.003.patch
>
>
> HDDS-443 provides a reusable progress bar to make it easier to add more freon
> tests, but the existing RandomKeyGenerator test
> (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java)
> still doesn't use it.
> It would be good to switch to use the new progress bar there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[
https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673153#comment-16673153
]
Zsolt Horvath edited comment on HDDS-692 at 11/2/18 2:21 PM:
-
{{I'm trying to resubmit the patch with these local results:}}
{{INFO] ---}}
{{ [INFO] T E S T S}}
{{ [INFO] ---}}
{{ [INFO] Running org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
14.896 s - in org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Running
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.402 s - in
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Running org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.066 s - in org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeRestart}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.518 s - in org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.368 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestDataValidate}}
{{ [INFO] Running
org.apache.hadoop.ozone.freon.TestFreonWithDatanodeFastRestart}}
was (Author: horzsolt2006):
{{I'm trying to resubmit the patch with these local results:}}
{{INFO] ---}}
{{ [INFO] T E S T S}}
{{ [INFO] ---}}
{{ [INFO] Running org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
14.896 s - in org.apache.hadoop.ozone.scm.TestContainerSQLCli}}
{{ [INFO] Running
org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402
s - in org.apache.hadoop.ozone.genconf.TestGenerateOzoneRequiredConfigurations}}
{{ [INFO] Running org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.066 s - in org.apache.hadoop.ozone.om.TestOmSQLCli}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestFreonWithDatanodeRestart}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.518
s - in org.apache.hadoop.ozone.freon.TestProgressBar}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.368 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator}}
{{ [INFO] Running org.apache.hadoop.ozone.freon.TestDataValidate}}
{{ [INFO] Running
org.apache.hadoop.ozone.freon.TestFreonWithDatanodeFastRestart}}
> Use the ProgressBar class in the RandomKeyGenerator freon test
> --
>
> Key: HDDS-692
> URL: https://issues.apache.org/jira/browse/HDDS-692
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Tools
>Reporter: Elek, Marton
>Assignee: Zsolt Horvath
>Priority: Major
> Attachments: HDDS-692.001.patch, HDDS-692.002.patch,
> HDDS-692.003.patch
>
>
> HDDS-443 provides a reusable progress bar to make it easier to add more freon
> tests, but the existing RandomKeyGenerator test
> (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java)
> still doesn't use it.
> It would be good to switch to use the new progress bar there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-692) Use the ProgressBar class in the RandomKeyGenerator freon test
[ https://issues.apache.org/jira/browse/HDDS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Horvath updated HDDS-692: --- Status: Patch Available (was: Open) > Use the ProgressBar class in the RandomKeyGenerator freon test > -- > > Key: HDDS-692 > URL: https://issues.apache.org/jira/browse/HDDS-692 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Tools >Reporter: Elek, Marton >Assignee: Zsolt Horvath >Priority: Major > Attachments: HDDS-692.001.patch, HDDS-692.002.patch, > HDDS-692.003.patch > > > HDDS-443 provides a reusable progress bar to make it easier to add more freon > tests, but the existing RandomKeyGenerator test > (hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java) > still doesn't use it. > It would be good to switch to use the new progress bar there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDDS-796:
---
Status: Patch Available (was: Open)
For Java Optional, if the value is not present, NoSuchElementException will be
returned. So the error was happened. Making a improvement for this: do the
additional check and return the correct host name.
Attach the patch.
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
>
> Key: HDDS-796
> URL: https://issues.apache.org/jira/browse/HDDS-796
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-796.001.patch
>
>
> After replacing usage of Guava Optional with Java Optional (HDDS-751), there
> are two unit test failures generated
> ([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
> unit tests are not fully triggered and tested under HDDS-751, :P
> {noformat}
> java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
> at
> org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14048:
---
Status: Patch Available (was: In Progress)
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-788) Change title page of bucket browser in S3gateway
[ https://issues.apache.org/jira/browse/HDDS-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673255#comment-16673255 ] Hudson commented on HDDS-788: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15347 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15347/]) HDDS-788. Change title page of bucket browser in S3gateway. Contributed (bharat: rev 6d9c18cfa9a458423b832a59166a15d098281ccd) * (edit) hadoop-ozone/s3gateway/src/main/resources/browser.html * (add) hadoop-ozone/s3gateway/src/main/resources/webapps/static/images/ozone.ico * (edit) hadoop-ozone/s3gateway/src/main/resources/webapps/s3gateway/WEB-INF/web.xml > Change title page of bucket browser in S3gateway > > > Key: HDDS-788 > URL: https://issues.apache.org/jira/browse/HDDS-788 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Fix For: 0.3.0, 0.4.0 > > Attachments: HDDS-788.00.patch, HDDS-788.01.patch > > > We need to change the name of bucket browser to Ozone S3 Explorer. > And also change title bar image to ozone icon. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Description: Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) was: Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for
[jira] [Created] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
Yiqun Lin created HDDS-796:
--
Summary: Fix failed test
TestStorageContainerManagerHttpServer#testHttpPolicy
Key: HDDS-796
URL: https://issues.apache.org/jira/browse/HDDS-796
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin
After replacing usage of Guava Optional with Java Optional (HDDS-751), there
are two unit test failures generated
([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
unit tests are not fully triggered and tested under HDDS-751, :P
{noformat}
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
at
org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
at
org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
at
org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-796) Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
[
https://issues.apache.org/jira/browse/HDDS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDDS-796:
---
Attachment: HDDS-796.001.patch
> Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy
>
>
> Key: HDDS-796
> URL: https://issues.apache.org/jira/browse/HDDS-796
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDDS-796.001.patch
>
>
> After replacing usage of Guava Optional with Java Optional (HDDS-751), there
> are two unit test failures generated
> ([https://builds.apache.org/job/PreCommit-HDDS-Build/1594/testReport/]). The
> unit tests are not fully triggered and tested under HDDS-751, :P
> {noformat}
> java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getBindAddress(BaseHttpServer.java:121)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.getHttpBindAddress(BaseHttpServer.java:148)
> at
> org.apache.hadoop.hdds.server.BaseHttpServer.(BaseHttpServer.java:63)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerHttpServer.(StorageContainerManagerHttpServer.java:34)
> at
> org.apache.hadoop.hdds.scm.TestStorageContainerManagerHttpServer.testHttpPolicy(TestStorageContainerManagerHttpServer.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673168#comment-16673168
]
Erik Krogen commented on HDFS-14048:
The fix is simple: when clearing the exception state, also do a
{{lastException.clear()}}.
[~szetszwo], any interest in helping to review based on involvement with past
JIRAs?
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14017) ObserverReadProxyProviderWithIPFailover should work with HA configuration
[
https://issues.apache.org/jira/browse/HDFS-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673267#comment-16673267
]
Erik Krogen commented on HDFS-14017:
Hey [~vagarychen], as I understand it, this patch essentially removes all
IPFailover-based functionality like automatically going to active rather than
searching for it. You've explicitly replaced the IPFailoverProxyProvider with a
version of a ConfiguredFailoverProxyProvider. It's basically just an
ObserverReadProxyProvider, but loading its configs in a hacky manner instead of
specifying them correctly. I can't see any situation in which it would be
better to use this PP than the existing ORPP. Please let me know if I'm missing
anything here, though.
I think the right way to handle this is to continue using IPFailoverPP for the
active, but add a way to _additionally_ configure the full set of NameNodes for
the ORPPWithIPFailover to use. This maintains as much existing functionality as
possible, and only falls back to the scanning mechanism for new functionality.
> ObserverReadProxyProviderWithIPFailover should work with HA configuration
> -
>
> Key: HDFS-14017
> URL: https://issues.apache.org/jira/browse/HDFS-14017
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-14017-HDFS-12943.001.patch,
> HDFS-14017-HDFS-12943.002.patch
>
>
> Currently {{ObserverReadProxyProviderWithIPFailover}} extends
> {{ObserverReadProxyProvider}}, and the only difference is changing the proxy
> factory to use {{IPFailoverProxyProvider}}. However this is not enough
> because when calling constructor of {{ObserverReadProxyProvider}} in
> super(...), the follow line:
> {code:java}
> nameNodeProxies = getProxyAddresses(uri,
> HdfsClientConfigKeys.DFS_NAMENODE_RPC_ADDRESS_KEY);
> {code}
> will try to resolve the all configured NN addresses to do configured
> failover. But in the case of IPFailover, this does not really apply.
>
> A second issue closely related is about delegation token. For example, in
> current IPFailover setup, say we have a virtual host nn.xyz.com, which points
> to either of two physical nodes nn1.xyz.com or nn2.xyz.com. In current HDFS,
> there is always only one DT being exchanged, which has hostname nn.xyz.com.
> Server only issues this DT, and client only knows the host nn.xyz.com, so all
> is good. But in Observer read, even with IPFailover, the client will no
> longer contacting nn.xyz.com, but will actively reaching to nn1.xyz.com and
> nn2.xyz.com. During this process, current code will look for DT associated
> with hostname nn1.xyz.com or nn2.xyz.com, which is different from the DT
> given by NN. causing Token authentication to fail. This happens in
> {{AbstractDelegationTokenSelector#selectToken}}. New IPFailover proxy
> provider will need to resolve this as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673461#comment-16673461
]
Hadoop QA commented on HDFS-14048:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14048 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946690/HDFS-14048.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ac698ebdc889 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (HDFS-14024) RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService
[
https://issues.apache.org/jira/browse/HDFS-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673470#comment-16673470
]
Íñigo Goiri commented on HDFS-14024:
Thank you!
> RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService
> -
>
> Key: HDFS-14024
> URL: https://issues.apache.org/jira/browse/HDFS-14024
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-14024-HDFS-13891.0.patch, HDFS-14024.0.patch
>
>
> Routers may be proxying for a downstream name node that is NOT migrated to
> understand "ProvidedCapacityTotal". updateJMXParameters method in
> NamenodeHeartbeatService should handle this without breaking.
>
> {code:java}
> jsonObject.getLong("MissingBlocks"),
> jsonObject.getLong("PendingReplicationBlocks"),
> jsonObject.getLong("UnderReplicatedBlocks"),
> jsonObject.getLong("PendingDeletionBlocks"),
> jsonObject.getLong("ProvidedCapacityTotal"));
> {code}
> One way to do this is create a json wrapper while gives back some default if
> json node is not found.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673446#comment-16673446
]
Erik Krogen edited comment on HDFS-14048 at 11/2/18 5:22 PM:
-
Thanks for taking a look [~jojochuang]! Actually, the error-prone code path is
when {{createBlockOutputStream()}} is called by {{setupPipelineInternal()}} (in
turn called by {{setupPipelineForAppendOrRecovery()}} -- in this case it is the
recovery situation). So it is actually only if the restart happens on the
_last_ block of the file, else the new block creation will clear the exception
as you described.
was (Author: xkrogen):
Thanks for taking a look [~jojochuang]! Actually, the error-prone code path is
when {{createBlockOutputStream()}} is called by {{setupPipelineInternal()}} (in
turn called by {{setupPipelineForAppendOrRecovery()}} -- in this case it is the
recovery situation)
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673446#comment-16673446
]
Erik Krogen commented on HDFS-14048:
Thanks for taking a look [~jojochuang]! Actually, the error-prone code path is
when {{createBlockOutputStream()}} is called by {{setupPipelineInternal()}} (in
turn called by {{setupPipelineForAppendOrRecovery()}} -- in this case it is the
recovery situation)
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13834) RBF: Connection creator thread should catch Throwable
[
https://issues.apache.org/jira/browse/HDFS-13834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673462#comment-16673462
]
Íñigo Goiri commented on HDFS-13834:
The only issue I see with [^HDFS-13834-HDFS-13891.3.patch] is that it is
potentially an infinite loop.
I don't think that would happen but I would prefer being careful.
Maybe using some of the generic test utils.
> RBF: Connection creator thread should catch Throwable
> -
>
> Key: HDFS-13834
> URL: https://issues.apache.org/jira/browse/HDFS-13834
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Critical
> Attachments: HDFS-13834-HDFS-13891.0.patch,
> HDFS-13834-HDFS-13891.1.patch, HDFS-13834-HDFS-13891.2.patch,
> HDFS-13834-HDFS-13891.3.patch, HDFS-13834.0.patch, HDFS-13834.1.patch
>
>
> Connection creator thread is a single thread thats responsible for creating
> all downstream namenode connections.
> This is very critical thread and hence should not die understand
> exception/error scenarios.
> We saw this behavior in production systems where the thread died leaving the
> router process in bad state.
> The thread should also catch a generic error/exception.
> {code}
> @Override
> public void run() {
> while (this.running) {
> try {
> ConnectionPool pool = this.queue.take();
> try {
> int total = pool.getNumConnections();
> int active = pool.getNumActiveConnections();
> if (pool.getNumConnections() < pool.getMaxSize() &&
> active >= MIN_ACTIVE_RATIO * total) {
> ConnectionContext conn = pool.newConnection();
> pool.addConnection(conn);
> } else {
> LOG.debug("Cannot add more than {} connections to {}",
> pool.getMaxSize(), pool);
> }
> } catch (IOException e) {
> LOG.error("Cannot create a new connection", e);
> }
> } catch (InterruptedException e) {
> LOG.error("The connection creator was interrupted");
> this.running = false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14045) Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN
[ https://issues.apache.org/jira/browse/HDFS-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673469#comment-16673469 ] Íñigo Goiri commented on HDFS-14045: I think this is pretty useful, specially with the observer effort (CC [~xkrogen] for awareness). I would also be more comfortable having some unit test. > Use different metrics in DataNode to better measure latency of > heartbeat/blockReports/incrementalBlockReports of Active/Standby NN > -- > > Key: HDFS-14045 > URL: https://issues.apache.org/jira/browse/HDFS-14045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Attachments: HDFS-14045.001.patch > > > Currently DataNode uses same metrics to measure rpc latency of NameNode, but > Active and Standby usually have different performance at the same time, > especially in large cluster. For example, rpc latency of Standby is very long > when Standby is catching up editlog. We may misunderstand the state of HDFS. > Using different metrics for Active and standby can help us obtain more > precise metric data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-791) Support Range header for ozone s3 object download
[
https://issues.apache.org/jira/browse/HDDS-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-791:
Summary: Support Range header for ozone s3 object download (was: Support
Range header for Oozne s3 object download)
> Support Range header for ozone s3 object download
> -
>
> Key: HDDS-791
> URL: https://issues.apache.org/jira/browse/HDDS-791
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: S3
>Reporter: Elek, Marton
>Priority: Major
>
> Using s3 rest api smaller chunks of an object could be uploaded with using
> Range headers:
> For example:
> {code}
> GET /example-object HTTP/1.1
> Host: example-bucket.s3.amazonaws.com
> x-amz-date: Fri, 28 Jan 2011 21:32:02 GMT
> Range: bytes=0-9
> Authorization: AWS AKIAIOSFODNN7EXAMPLE:Yxg83MZaEgh3OZ3l0rLo5RTX11o=
> Sample Response with Specified Range of the Object Bytes
> {code}
> Can be implemented with using the seek method on OzoneInputStream.
> The Range header support is one of the missing piece for fully support s3a
> interface.
> References:
> Range header spec:
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
> Aws s3 doc:
> https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
[
https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673395#comment-16673395
]
Wei-Chiu Chuang commented on HDFS-14048:
LGTM. Don't know if there are existing tests that assumes the second close()
throws exception.
Also note nextBlockOutputStream() clears lastException before calling
createBlockOutputStream(). So I guess this only happens for the first block of
a file?
> DFSOutputStream close() throws exception on subsequent call after DataNode
> restart
> --
>
> Key: HDFS-14048
> URL: https://issues.apache.org/jira/browse/HDFS-14048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14048.000.patch
>
>
> We recently discovered an issue in which, during a rolling upgrade, some jobs
> were failing with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting:
> DatanodeInfoWithStorage[1.1.1.1:71,BP-,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing
> after waiting for the DataNode restart. We eventually realized that in some
> cases {{DFSOutputStream#close()}} may be called more than once, and that if
> so, the {{IOException}} above is thrown on the _second_ call to {{close()}}
> (this is even with HDFS-5335; prior to this it would have been thrown on all
> calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the
> new output stream is created, it resets the error states:
> {code}
> errorState.resetInternalError();
> // remove all restarting nodes from failed nodes list
> failed.removeAll(restartingNodes);
> restartingNodes.clear();
> {code}
> But it forgets to clear {{lastException}}. When
> {{DFSOutputStream#closeImpl()}} is called a second time, this block is
> triggered:
> {code}
> if (isClosed()) {
> LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
> closed, getStreamer().streamerClosed());
> try {
> getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and
> the "Datanode is restarting" exception is thrown even though the stream has
> already been successfully closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Attachment: HDFS-14047.001.patch > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch, HDFS-14047.001.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14030) Add query whether storage has been initialized to DN's JMXBean
[
https://issues.apache.org/jira/browse/HDFS-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673612#comment-16673612
]
Wei-Chiu Chuang commented on HDFS-14030:
Actually, this error is generated by querying /jmx, and the queries are sent by
monitoring systems like Cloudera Manager sends these requests regardless the
state of DataNode.
> Add query whether storage has been initialized to DN's JMXBean
> --
>
> Key: HDFS-14030
> URL: https://issues.apache.org/jira/browse/HDFS-14030
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: HDFS-14030.001.patch
>
>
> During the short period while the DN is starting and the storage is not yet
> available we get the following errors:
> {noformat}
> ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute VolumeInfo of
> Hadoop:service=DataNode,name=DataNodeInfo threw an exception
> javax.management.RuntimeMBeanException: java.lang.NullPointerException:
> Storage not yet initialized
> {noformat}
> I suggest to add an option to the DataNodeMXBean to query whether the storage
> has been initialized or not to surpass the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
