[jira] [Work logged] (MAPREDUCE-7270) TestHistoryViewerPrinter could be failed when the locale isn't English.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7270?focusedWorklogId=570258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-570258 ] ASF GitHub Bot logged work on MAPREDUCE-7270: - Author: ASF GitHub Bot Created on: 23/Mar/21 06:23 Start Date: 23/Mar/21 06:23 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #1942: URL: https://github.com/apache/hadoop/pull/1942#issuecomment-804651653 Thank you very much @ayushtkn The report was not posted here, but the new run was fine. Since there is a checkstyle warning, could you rebase this PR from `trunk` branch, and also fixe the checkstyle warning @sungpeo ? > ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java:58: Locale.setDefault(DEFAULT_LOCALE);: 'method def' child has incorrect indentation level 6, expected level should be 4. [Indentation] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 570258) Time Spent: 1h 20m (was: 1h 10m) > TestHistoryViewerPrinter could be failed when the locale isn't English. > --- > > Key: MAPREDUCE-7270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Sungpeo Kook >Assignee: Sungpeo Kook >Priority: Minor > Labels: pull-request-available > Attachments: MAPREDUCE-7270.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Both of testHumanPrinter and testHumanPrinterAll have expected string for > assertion. > But the actual result includes the Dateformat which can be different depends > on Locale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7330) Add access check for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie reassigned MAPREDUCE-7330: Assignee: lujie > Add access check for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Assignee: lujie >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated MAPREDUCE-7325:
---
Fix Version/s: 3.2.3
2.10.2
3.3.1
3.1.5
3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the contribution [~ahussein]! I have committed this to trunk -
branch-2.10.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7330) Add access check for getJobAttempts
[
https://issues.apache.org/jira/browse/MAPREDUCE-7330?focusedWorklogId=569794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-569794
]
ASF GitHub Bot logged work on MAPREDUCE-7330:
-
Author: ASF GitHub Bot
Created on: 22/Mar/21 15:31
Start Date: 22/Mar/21 15:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2798:
URL: https://github.com/apache/hadoop/pull/2798#issuecomment-804154415
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 36s | | hadoop-mapreduce-client-hs in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 75m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2798/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2798 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 074616c92ec2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ec0f550bd60ab6579cd06f253e922b3f53ea5521 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2798/1/testReport/ |
| Max. process+thread count | 727 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs |
|
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306260#comment-17306260
]
Jim Brennan commented on MAPREDUCE-7325:
Thanks for the fix [~ahussein]! +1 This change looks good to me. I will
commit later today.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7330) Add access check for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?focusedWorklogId=569731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-569731 ] ASF GitHub Bot logged work on MAPREDUCE-7330: - Author: ASF GitHub Bot Created on: 22/Mar/21 14:14 Start Date: 22/Mar/21 14:14 Worklog Time Spent: 10m Work Description: lujiefsi opened a new pull request #2798: URL: https://github.com/apache/hadoop/pull/2798 Hi, i have found an inconsist, all the methods who get job infomation have access check, except getJobAttempts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 569731) Remaining Estimate: 0h Time Spent: 10m > Add access check for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add access check for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated MAPREDUCE-7330: -- Labels: pull-request-available (was: ) > Add access check for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add access check for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Priority: Major (was: Critical) > Add access check for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Major > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add access check for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Summary: Add access check for getJobAttempts (was: Add check access for getJobAttempts) > Add access check for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Critical > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add check access for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Attachment: (was: MAPREDUCE-7330_1.patch) > Add check access for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Critical > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add check access for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Description: Hi, i have found an inconsist, all the methods getting job infomation own access check, except getJobAttempts. I will fix it. (was: Hi, i have found an inconsist, all the method to get job infomation in cd ) > Add check access for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Critical > > Hi, i have found an inconsist, all the methods getting job infomation own > access check, except getJobAttempts. I will fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add check access for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Description: Hi, i have found an inconsist, all the method to get job infomation in cd (was: current, we do not check the access for getJobAttempts) > Add check access for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Critical > Attachments: MAPREDUCE-7330_1.patch > > > Hi, i have found an inconsist, all the method to get job infomation in cd -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add check access for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Priority: Critical (was: Major) > Add check access for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: lujie >Priority: Critical > Attachments: MAPREDUCE-7330_1.patch > > > current, we do not check the access for getJobAttempts -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7330) Add check access for getJobAttempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated MAPREDUCE-7330: - Issue Type: Bug (was: New Feature) > Add check access for getJobAttempts > > > Key: MAPREDUCE-7330 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7330 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: lujie >Priority: Critical > Attachments: MAPREDUCE-7330_1.patch > > > current, we do not check the access for getJobAttempts -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306172#comment-17306172
]
Ahmed Hussein commented on MAPREDUCE-7325:
--
[~Jim_Brennan] Can you please take a look at the patch?
The bug was due to an incorrect seek in the local fetcher. It works only if
encryption was disabled.
The changes are:
# The bug fix is just two lines change in LocalFetcher.java
# In {{TestMRIntermediateDataEncryption}}, I added an extra step to verify the
output of the jobs. This is done by comparing the checksum of the output files.
The reference checksum comes from a job that starts at the beginning of the
unit test with encryption disabled.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
