[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=794605=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-794605 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 24/Jul/22 06:11 Start Date: 24/Jul/22 06:11 Worklog Time Spent: 10m Work Description: virajjasani closed pull request #4461: HADOOP-18303. Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime URL: https://github.com/apache/hadoop/pull/4461 Issue Time Tracking --- Worklog Id: (was: 794605) Time Spent: 3h 20m (was: 3h 10m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784395 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 24/Jun/22 02:13 Start Date: 24/Jun/22 02:13 Worklog Time Spent: 10m Work Description: virajjasani commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1165109458 Even if HADOOP-18033 had not excluded javax.ws.rs-api from shading, as [mentioned here](https://github.com/apache/hadoop/pull/4461#pullrequestreview-1012678699), we would have anyways caused subtle problems by having only one version of common class in the final client jar (as opposed to having different classes for both javax.ws.rs-api and jsr311-api). Issue Time Tracking --- Worklog Id: (was: 784395) Time Spent: 3h 10m (was: 3h) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784392 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 24/Jun/22 02:08 Start Date: 24/Jun/22 02:08 Worklog Time Spent: 10m Work Description: virajjasani commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1165107308 While HADOOP-18033 is released in 3.3.2, HADOOP-18178 has also made it's way to 3.3.3. Issue Time Tracking --- Worklog Id: (was: 784392) Time Spent: 3h (was: 2h 50m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784376 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 23/Jun/22 23:13 Start Date: 23/Jun/22 23:13 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-116433 Yeps, if people let us do that, that jira is already in a released version Issue Time Tracking --- Worklog Id: (was: 784376) Time Spent: 2h 50m (was: 2h 40m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784342 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 23/Jun/22 19:37 Start Date: 23/Jun/22 19:37 Worklog Time Spent: 10m Work Description: sunchao commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1164795939 In that case should we consider reverting it for now until we are ready to upgrade both jersey and jackson together at some point? Issue Time Tracking --- Worklog Id: (was: 784342) Time Spent: 2h 40m (was: 2.5h) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784338 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 23/Jun/22 19:21 Start Date: 23/Jun/22 19:21 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1164781602 Both shading and the issue that I am talking about are due to HADOOP-18033 Jackson upgrade added a rs-api jar which is not shaded and which all cause conflicts in Tez jsr311 jar already present Issue Time Tracking --- Worklog Id: (was: 784338) Time Spent: 2.5h (was: 2h 20m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=784332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784332 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 23/Jun/22 18:47 Start Date: 23/Jun/22 18:47 Worklog Time Spent: 10m Work Description: sunchao commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1164752328 > @sunchao It doesn't, that is what I was saying, even if we sort the shading issue, the Tez issue is gonna stay because it doesn't use the shaded Jar and gets both these jars in classpath and things mess up. I see. This looks like a separate issue from this PR, is that correct? is this because of https://issues.apache.org/jira/browse/HADOOP-18033? The issue seems tricky though. I wonder if we can shade jersey 2 in `hadoop-thirdparty` and then update Hadoop to use that. Issue Time Tracking --- Worklog Id: (was: 784332) Time Spent: 2h 20m (was: 2h 10m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783827 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 22/Jun/22 12:01 Start Date: 22/Jun/22 12:01 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1163011238 bq. Looks like upgrading jersey to get rid of jsr311-api is the only solution then. ok, but then no aspects of jersey 2 must be unshaded or every project which still uses 1.19 is going to break. we don't want to do that Issue Time Tracking --- Worklog Id: (was: 783827) Time Spent: 2h 10m (was: 2h) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783793 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 22/Jun/22 09:46 Start Date: 22/Jun/22 09:46 Worklog Time Spent: 10m Work Description: pan3793 commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1162886176 Looks like upgrading jersey to get rid of `jsr311-api` is the only solution then. Issue Time Tracking --- Worklog Id: (was: 783793) Time Spent: 2h (was: 1h 50m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783734 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 22/Jun/22 07:23 Start Date: 22/Jun/22 07:23 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1162743798 @sunchao It doesn't, that is what I was saying, even if we sort the shading issue, the Tez issue is gonna stay because it doesn't use the shaded Jar and gets both these jars in classpath and things mess up. Can check the exception in the end here: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-213/4/testReport/org.apache.tez.history/TestHistoryParser/testParserWithFailedJob/ Issue Time Tracking --- Worklog Id: (was: 783734) Time Spent: 1h 50m (was: 1h 40m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783732 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 22/Jun/22 07:13 Start Date: 22/Jun/22 07:13 Worklog Time Spent: 10m Work Description: sunchao commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1162735000 In this case, the problematic class is `org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException` which is only in `javax.ws.rs:javax.ws.rs-api:jar` and is not shaded before this PR, and hence the exception. > Let us assume we did something with the shading, but Projects like Tez, when they add Hadoop as dependency they get both Jars and both have similar classes, so that creates a mess, something similar error message like here ... Hmm why is that? does Hive/Tez use shaded Hadoop client? the shading also comes with relocation so that it shouldn't interfere with the other dependencies. Issue Time Tracking --- Worklog Id: (was: 783732) Time Spent: 1h 40m (was: 1.5h) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783497 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 21/Jun/22 17:52 Start Date: 21/Jun/22 17:52 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1162099605 @pan3793 might be good to answer the kyuubi & spark issue is flagged by him here: https://github.com/apache/hadoop/pull/3764#issuecomment-1158641569 and some details here: https://issues.apache.org/jira/browse/HADOOP-18033?focusedCommentId=17555953=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17555953 @pan3793 can you please share some more details on that... Issue Time Tracking --- Worklog Id: (was: 783497) Time Spent: 1.5h (was: 1h 20m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783488 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 21/Jun/22 17:28 Start Date: 21/Jun/22 17:28 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1162057008 > kirby & spark are crying due to missing classes. so why aren't they explicitly asking for the jars? or are some bits of our code are still asking for things by their unshaded classnames? Issue Time Tracking --- Worklog Id: (was: 783488) Time Spent: 1h 20m (was: 1h 10m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783344 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 21/Jun/22 11:46 Start Date: 21/Jun/22 11:46 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1161637780 >I'm getting really confused now. Before this PR, conflicting imports were both being shaded. Now they are being merged in unshaded and the issue is which ones not to shade? I am also confused No, one was shaded one wasn't, the one not shaded is getting added here because kirby & spark are crying due to missing classes. The question is yes which one to shade and which one not to, In Ideal situation only one these jars should be there because they provide similar classes and to be used by different version. In our case Jackson use the new Jar, and Jersey uses the old Jar, so we need both Jar in hadoop. If we upgrade Jersey also, both Jackson & Jersey will need the new jar and we can remove one jar and this conflict. Let us assume we did something with the shading, but Projects like Tez, when they add Hadoop as dependency they get both Jars and both have similar classes, so that creates a mess, something similar error message like here: https://stackoverflow.com/questions/23277429/java-lang-abstractmethoderror-javax-ws-rs-core-uribuilder-uri/26767488#26767488 Regarding Solution: I don't have a perfect solution, and definitely need pointers, Not sure what we can do with shading, if we figure out may be spark/kriby may be happy. But if we exclude the old jar who will be unhappy god knows Shading won't solve Hive/Tez issue Jersey upgrade: Not sure we can do for a maintenance release, people have objections for protobuf as well, and post that also I guess downstream also have to upgrade Jersey, that won't be easy either. That is all what I know or can think of, Not sure what all is Right/Wrong. Hope I didn't contribute more to the confusion. Issue Time Tracking --- Worklog Id: (was: 783344) Time Spent: 1h 10m (was: 1h) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783292 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 21/Jun/22 09:44 Start Date: 21/Jun/22 09:44 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1161514464 i wouldn't trust the ordering of the merge. could one jar be excluded explicitly. I'm getting really confused now. Before this PR, conflicting imports were both being shaded. Now they are being merged in unshaded and the issue is *which ones not to shade?* Issue Time Tracking --- Worklog Id: (was: 783292) Time Spent: 1h (was: 50m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783287 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 21/Jun/22 09:42 Start Date: 21/Jun/22 09:42 Worklog Time Spent: 10m Work Description: steveloughran commented on code in PR #4461: URL: https://github.com/apache/hadoop/pull/4461#discussion_r902399389 ## pom.xml: ## @@ -120,7 +120,6 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 3.10.1 bash - Review Comment: aah Issue Time Tracking --- Worklog Id: (was: 783287) Time Spent: 50m (was: 40m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[
https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783123
]
ASF GitHub Bot logged work on HADOOP-18303:
---
Author: ASF GitHub Bot
Created on: 20/Jun/22 19:39
Start Date: 20/Jun/22 19:39
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4461:
URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1160787124
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 59m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | shadedclient | 21m 29s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 87m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4461/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4461 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux ee8e2b0f7432 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3f6c6e0fb05e23c3c6ac4b30e4431ecd191e8db0 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4461/6/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-client-modules/hadoop-client-runtime U:
hadoop-client-modules/hadoop-client-runtime |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4461/6/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783108 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 20/Jun/22 18:06 Start Date: 20/Jun/22 18:06 Worklog Time Spent: 10m Work Description: virajjasani commented on code in PR #4461: URL: https://github.com/apache/hadoop/pull/4461#discussion_r901912030 ## pom.xml: ## @@ -120,7 +120,6 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 3.10.1 bash - Review Comment: Sure thing, this was done just to trigger the full test run (and hence we see hdfs javadoc failure because all modules were involved in the build runs) Issue Time Tracking --- Worklog Id: (was: 783108) Time Spent: 0.5h (was: 20m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=783010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783010 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 20/Jun/22 14:44 Start Date: 20/Jun/22 14:44 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4461: URL: https://github.com/apache/hadoop/pull/4461#issuecomment-1160537246 hdfs one is my fault :( Issue Time Tracking --- Worklog Id: (was: 783010) Time Spent: 20m (was: 10m) > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18303) Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
[ https://issues.apache.org/jira/browse/HADOOP-18303?focusedWorklogId=782978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-782978 ] ASF GitHub Bot logged work on HADOOP-18303: --- Author: ASF GitHub Bot Created on: 20/Jun/22 13:20 Start Date: 20/Jun/22 13:20 Worklog Time Spent: 10m Work Description: steveloughran commented on code in PR #4461: URL: https://github.com/apache/hadoop/pull/4461#discussion_r901654345 ## pom.xml: ## @@ -120,7 +120,6 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 3.10.1 bash - Review Comment: cut this change; i want the most minimal backport possible Issue Time Tracking --- Worklog Id: (was: 782978) Remaining Estimate: 0h Time Spent: 10m > Remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > -- > > Key: HADOOP-18303 > URL: https://issues.apache.org/jira/browse/HADOOP-18303 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > > As part of HADOOP-18033, we have excluded shading of javax.ws.rs-api from > both hadoop-client-runtime and hadoop-client-minicluster. This has caused > issues for downstreamers e.g. > [https://github.com/apache/incubator-kyuubi/issues/2904], more discussions. > We should put the shading back in hadoop-client-runtime to fix CNFE issues > for downstreamers. > cc [~ayushsaxena] [~pan3793] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
