[jira] [Comment Edited] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390092#comment-15390092
]
Haibo Chen edited comment on MAPREDUCE-6741 at 7/22/16 7:43 PM:
Good point! I have not thought about updating the jobconf file in HDFS. A
question that may or may not be relevant to this. I notice that for every job
there are always two job conf files. One file, named job.xml, is submitted by
the client and localized as task container resource. The other, named
$(jobid)_conf.xml, is created by JobHistoryEventHandler and loaded by JHS. Any
particular reason why we have two files lying around for one job?
was (Author: haibochen):
Good point! I have not thought about updating the jobconf file in HDFS. A
question that may or may not be relevant to this. I notice that for every job
there are always two job conf files. One file, named job.xml, is submitted by
the client and localized as task container resource. The other, named
{jobid}_conf.xml, is created by JobHistoryEventHandler and loaded by JHS. Any
particular reason why we have two files lying around for one job?
> add JHS support to hide job conf properties from Web UI
> ---
>
> Key: MAPREDUCE-6741
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> JHS today displays all Job conf properties in Web UI directly. Users may have
> some credentials or any sensitive information they added to the job conf but
> do not want to be shown in Web UI. It'd be nice if we can allow users to
> specify a set of properties which JHS will filter out when Job conf is
> displayed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390092#comment-15390092
]
Haibo Chen commented on MAPREDUCE-6741:
---
Good point! I have not thought about updating the jobconf file in HDFS. A
question that may or may not be relevant to this. I notice that for every job
there are always two job conf files. One file, named job.xml, is submitted by
the client and localized as task container resource. The other, named
{jobid}_conf.xml, is created by JobHistoryEventHandler and loaded by JHS. Any
particular reason why we have two files lying around for one job?
> add JHS support to hide job conf properties from Web UI
> ---
>
> Key: MAPREDUCE-6741
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> JHS today displays all Job conf properties in Web UI directly. Users may have
> some credentials or any sensitive information they added to the job conf but
> do not want to be shown in Web UI. It'd be nice if we can allow users to
> specify a set of properties which JHS will filter out when Job conf is
> displayed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390040#comment-15390040
]
Hadoop QA commented on MAPREDUCE-6744:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 136m 5s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMRCJCFileOutputCommitter |
| Timed out junit tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12819658/MAPREDUCE-6744.001.patch
|
| JIRA Issue | MAPREDUCE-6744 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ec2244c6b7dd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c60a68c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6638/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6638/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6638/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6638/console |
| Powe
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389986#comment-15389986 ] Gera Shegalov commented on MAPREDUCE-6724: -- [~haibochen], what I mean if you revert your change to hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java , and apply my shorter [change|https://issues.apache.org/jira/browse/MAPREDUCE-6724?focusedCommentId=15384827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15384827] to it, I think it tests the bugs you are fixing: it will fail without the fix, and succeed with the fix. > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > - > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, > mapreduce6724.003.patch, mapreduce6724.004.patch, mapreduce6724.005.patch, > mapreduce6724.006.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389967#comment-15389967 ] Haibo Chen commented on MAPREDUCE-6724: --- [~jira.shegalov] Can you elaborate a little big more on what you mean by existing test covers regression? My understanding of testLargeMemoryLimits() is that it checks memoryLimit and maxInMemReduceLimit, both of which can not be tested with verifyReserveLargeMapOutput(). I could restore changes made to testLargeMemoryLimits to minimize changes to existing tests. By "testing MAX_VALUE", do you mean that we need to have another test case in which singleSHuffleLimit is set to Integer.MAX_VAL? > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > - > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, > mapreduce6724.003.patch, mapreduce6724.004.patch, mapreduce6724.005.patch, > mapreduce6724.006.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389788#comment-15389788 ] Eric Badger commented on MAPREDUCE-6744: Linking JIRA that previously increased some (but not all) timeouts on DFSIO tests. > Increase timeout on TestDFSIO tests > --- > > Key: MAPREDUCE-6744 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6744 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6744.001.patch > > > The timeout on these tests is only 3 seconds and so one of the tests in the > suite will fail on a regular basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
Eric Badger created MAPREDUCE-6744: -- Summary: Increase timeout on TestDFSIO tests Key: MAPREDUCE-6744 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6744 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Eric Badger The timeout on these tests is only 3 seconds and so one of the tests in the suite will fail on a regular basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated MAPREDUCE-6744: --- Status: Patch Available (was: Open) > Increase timeout on TestDFSIO tests > --- > > Key: MAPREDUCE-6744 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6744 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6744.001.patch > > > The timeout on these tests is only 3 seconds and so one of the tests in the > suite will fail on a regular basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated MAPREDUCE-6744: --- Attachment: MAPREDUCE-6744.001.patch Attaching patch to increase timeout from 3 to 10 seconds. > Increase timeout on TestDFSIO tests > --- > > Key: MAPREDUCE-6744 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6744 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6744.001.patch > > > The timeout on these tests is only 3 seconds and so one of the tests in the > suite will fail on a regular basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6744) Increase timeout on TestDFSIO tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger reassigned MAPREDUCE-6744: -- Assignee: Eric Badger > Increase timeout on TestDFSIO tests > --- > > Key: MAPREDUCE-6744 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6744 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > > The timeout on these tests is only 3 seconds and so one of the tests in the > suite will fail on a regular basis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389744#comment-15389744 ] Jason Lowe commented on MAPREDUCE-6741: --- If that's the case, would it make more sense to have the AM redact the values when it writes out the jobconf that will be picked up by the JHS? Then even admins can't see it once the job is completed because there should be no more copies lying around with the original values, even in HDFS. > add JHS support to hide job conf properties from Web UI > --- > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389732#comment-15389732 ] Haibo Chen commented on MAPREDUCE-6741: --- I am thinking of redacting the values in the UI for all users to make it simple. This essentially comes from requests from customers who have legacy MR jobs that used S3/Azure as input&output. Instead of using CredentialProvider, they put their credentials directly in JobConf, which is then displayed directly JHS UI. Some of them are aware of job ACL, but also want to disallow queue admins or whoever are always given access to all jobs by JHS, to easily access their credentials. Redacting the credentials in UI doesn't really solve the security problem. The credentials are still stored in jobconf in HDFS after all. But it does make the credentials less exposed. > add JHS support to hide job conf properties from Web UI > --- > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6743) nativetask unit tests need to provide usable output
Allen Wittenauer created MAPREDUCE-6743: --- Summary: nativetask unit tests need to provide usable output Key: MAPREDUCE-6743 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 Project: Hadoop Map/Reduce Issue Type: Test Components: nativetask Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which provides an binary exit code to determine failure. This means there is no output generated by the Jenkins run to actually debug or provide hints as to what failed. Given that nttest is written with gtest, it should be configured to either spit out junit or TAP which can then be used to provide further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6742) Test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-6742. - Resolution: Not A Problem > Test > > > Key: MAPREDUCE-6742 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6742 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Prashanth G B > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389627#comment-15389627 ] Jason Lowe commented on MAPREDUCE-6741: --- Another requirements clarification: when the properties are to be hidden, will they be completely removed from the output or will the keys appear with redacted values? > add JHS support to hide job conf properties from Web UI > --- > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add JHS support to hide job conf properties from Web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389626#comment-15389626 ] Jason Lowe commented on MAPREDUCE-6741: --- Will this filtering be for all users or will it still show the properties for the owner of the job? > add JHS support to hide job conf properties from Web UI > --- > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6380: Assignee: Kai Sasaki (was: Varun Saxena) > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Assignee: Kai Sasaki >Priority: Trivial > Attachments: MAPREDUCE-6380.01.patch, MAPREDUCE-6380.02.patch, > MAPREDUCE-6380.03.patch, MAPREDUCE-6380.04.patch, MAPREDUCE-6380.05.patch, > MAPREDUCE-6380.06.patch, MAPREDUCE-6380.07.patch > > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned MAPREDUCE-6380: --- Assignee: Varun Saxena (was: Kai Sasaki) > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Assignee: Varun Saxena >Priority: Trivial > Attachments: MAPREDUCE-6380.01.patch, MAPREDUCE-6380.02.patch, > MAPREDUCE-6380.03.patch, MAPREDUCE-6380.04.patch, MAPREDUCE-6380.05.patch, > MAPREDUCE-6380.06.patch, MAPREDUCE-6380.07.patch > > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6742) Test
Prashanth G B created MAPREDUCE-6742: Summary: Test Key: MAPREDUCE-6742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6742 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Prashanth G B -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389209#comment-15389209
]
Brahma Reddy Battula commented on MAPREDUCE-6682:
-
[~ajisakaa] thanks for working on this issue and pinging me..
Approach overall looks great.
Some minor nits:
1) *{{TestMRCJCFileOutputCommitter}}:* As {{testCommitter}} also calling
{{FileUtil.fullyDelete(new File(outDir.toString()))}} at the end of
testcase,how about adding in teardown..?
2) *{{TestMRTimelineEventHandling}}:* do we need to handle {{inDir}} delete
also..?
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
