[jira] [Commented] (HDFS-14846) libhdfs tests are failing on trunk due to jni usage bugs
[
https://issues.apache.org/jira/browse/HDFS-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931909#comment-16931909
]
Anu Engineer commented on HDFS-14846:
-
Thank you for your contribution. I have committed this patch to the trunk
branch.
> libhdfs tests are failing on trunk due to jni usage bugs
>
>
> Key: HDFS-14846
> URL: https://issues.apache.org/jira/browse/HDFS-14846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> While working on HDFS-14564, I noticed that the libhdfs tests are failing on
> trunk (both on Hadoop QA and locally). I did some digging and found out that
> the {{-Xcheck:jni}} flag is causing a bunch of crashes. I haven't been able
> to pinpoint what caused this regression, but my best guess is that an upgrade
> in the JDK we use in Hadoop QA started causing these failures. I looked back
> at some old JIRAs and it looks like the tests work on Java 1.8.0_212, but
> Hadoop QA is running 1.8.0_222 (as is my local env) (I couldn't confirm this
> theory because I'm having trouble getting Java 1.8.0_212 installed next to
> 1.8.0_222 on my Ubuntu machine) (even after re-winding the commit history
> back to a known good commit where the libhdfs passed, the tests still fail,
> so I don't think a code change caused the regressions).
> The failures are a bunch of "FATAL ERROR in native method: Bad global or
> local ref passed to JNI" errors. After doing some debugging, it looks like
> {{-Xcheck:jni}} now errors out if any code tries to pass a local ref to
> {{DeleteLocalRef}} twice (previously it looked like it didn't complain) (we
> have some checks to avoid this, but it looks like they don't work as
> expected).
> There are a few places in the libhdfs code where this pattern causes a crash,
> as well as one place in {{JniBasedUnixGroupsMapping}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14846) libhdfs tests are failing on trunk due to jni usage bugs
[
https://issues.apache.org/jira/browse/HDFS-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931908#comment-16931908
]
Hudson commented on HDFS-14846:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17316 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17316/])
HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs
(aengineer: rev 3cf6e4272f192c69a161307ad9d35142c5a845c5)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
* (edit)
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/native_mini_dfs.c
> libhdfs tests are failing on trunk due to jni usage bugs
>
>
> Key: HDFS-14846
> URL: https://issues.apache.org/jira/browse/HDFS-14846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> While working on HDFS-14564, I noticed that the libhdfs tests are failing on
> trunk (both on Hadoop QA and locally). I did some digging and found out that
> the {{-Xcheck:jni}} flag is causing a bunch of crashes. I haven't been able
> to pinpoint what caused this regression, but my best guess is that an upgrade
> in the JDK we use in Hadoop QA started causing these failures. I looked back
> at some old JIRAs and it looks like the tests work on Java 1.8.0_212, but
> Hadoop QA is running 1.8.0_222 (as is my local env) (I couldn't confirm this
> theory because I'm having trouble getting Java 1.8.0_212 installed next to
> 1.8.0_222 on my Ubuntu machine) (even after re-winding the commit history
> back to a known good commit where the libhdfs passed, the tests still fail,
> so I don't think a code change caused the regressions).
> The failures are a bunch of "FATAL ERROR in native method: Bad global or
> local ref passed to JNI" errors. After doing some debugging, it looks like
> {{-Xcheck:jni}} now errors out if any code tries to pass a local ref to
> {{DeleteLocalRef}} twice (previously it looked like it didn't complain) (we
> have some checks to avoid this, but it looks like they don't work as
> expected).
> There are a few places in the libhdfs code where this pattern causes a crash,
> as well as one place in {{JniBasedUnixGroupsMapping}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14846) libhdfs tests are failing on trunk due to jni usage bugs
[
https://issues.apache.org/jira/browse/HDFS-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929293#comment-16929293
]
Sahil Takiar commented on HDFS-14846:
-
Hadoop QA looks good. [~jojochuang], [~smeng] could you take a look?
> libhdfs tests are failing on trunk due to jni usage bugs
>
>
> Key: HDFS-14846
> URL: https://issues.apache.org/jira/browse/HDFS-14846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> While working on HDFS-14564, I noticed that the libhdfs tests are failing on
> trunk (both on hadoop-yetus and locally). I dig some digging and found out
> that the {{-Xcheck:jni}} flag is causing a bunch of crashes. I haven't been
> able to pinpoint what caused this regression, but my best guess is that an
> upgrade in the JDK we use in hadoop-yetus started causing these failures. I
> looked back at some old JIRAs and it looks like the tests work on Java
> 1.8.0_212, but yetus is running 1.8.0_222 (as is my local env) (I couldn't
> confirm this theory because I'm having trouble getting install 1.8.0_212 next
> to 1.8.0_222 on my Ubuntu machine) (even after re-winding the commit history
> back to a known good commit where the libhdfs passed, the tests still fail,
> so I don't think a code change caused the regressions).
> The failures are a bunch of "FATAL ERROR in native method: Bad global or
> local ref passed to JNI" errors. After doing some debugging, it looks like
> {{-Xcheck:jni}} now errors out if any code tries to pass a local ref to
> {{DeleteLocalRef}} twice (previously it looked like it didn't complain) (we
> have some checks to avoid this, but it looks like they don't work as
> expected).
> There are a few places in the libhdfs code where this pattern causes a crash,
> as well as one place in {{JniBasedUnixGroupsMapping}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14846) libhdfs tests are failing on trunk due to jni usage bugs
[
https://issues.apache.org/jira/browse/HDFS-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928961#comment-16928961
]
Hadoop QA commented on HDFS-14846:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
48m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
32s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1436/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1436 |
| JIRA Issue | HDFS-14846 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 404d9186ce3c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 4852a90 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1436/1/testReport/ |
| Max. process+thread count | 1392 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-native-client U: . |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1436/1/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> libhdfs tests are failing on trunk due to jni usage bugs
>
>
> Key: HDFS-14846
> URL:
