[jira] [Commented] (MAPREDUCE-7293) All pages in JHS should honor yarn.webapp.filter-entity-list-by-user
[
https://issues.apache.org/jira/browse/MAPREDUCE-7293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181629#comment-17181629
]
Sunil G commented on MAPREDUCE-7293:
We were trying to block listing all jobs in the landing page of JHS if the
logged in user doesnt have any permissions.
In that case, user cannot access or aware about the specific job.
However if the user know how to construct the URL, i think its possible that
they can land on that page. We also need to see the cost of getting the user
check in individual page. In the list page, we could get all this from file
name hence it was fine. Other wise , we may need to scan file which will cause
performance issues. Kindly analyse and if there are no major perf issues, lets
do this.
Thanks
> All pages in JHS should honor yarn.webapp.filter-entity-list-by-user
>
>
> Key: MAPREDUCE-7293
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7293
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
>
> Currently only HsJobsBlock checks for the access. If user who doesn't have
> permission to access job page is able to do it which is wrong. So we need to
> have below check in HsJobBlock,HsTasksBlock and HsTaskPage
> {code:java}
> if (isFilterAppListByUserEnabled && ugi != null && !aclsManager
> .checkAccess(ugi, JobACL.VIEW_JOB, job.getUserName(), null)) {
>
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7292) Use snapshot diff to build file listing when copying to blob storage
[
https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181394#comment-17181394
]
Hadoop QA commented on MAPREDUCE-7292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
47s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
15s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
|
[jira] [Commented] (MAPREDUCE-7292) Use snapshot diff to build file listing when copying to blob storage
[
https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181371#comment-17181371
]
Hadoop QA commented on MAPREDUCE-7292:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 27m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 57m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
48s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
31s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
|
[jira] [Updated] (MAPREDUCE-7292) Use snapshot diff to build file listing when copying to blob storage
[ https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Liu updated MAPREDUCE-7292: Attachment: MAPREDUCE-7292.002.patch > Use snapshot diff to build file listing when copying to blob storage > > > Key: MAPREDUCE-7292 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7292 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: John Liu >Priority: Minor > Attachments: MAPREDUCE-7292.001.patch, MAPREDUCE-7292.002.patch > > > The DistCp sync option should be extensible for copying to blob storage, > which is not a distributed filesystem. Clients of DistCp could benefit from > using the HDFS snapshot diff report to create the file listing in less time. > A valid use case is to copy new files added to HDFS to a remote blob storage. > The client ensures all new files are copied over but does not require the > destination filesystem to be a distributed filesystem or have the previous > snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7292) Use snapshot diff to build file listing when copying to blob storage
[ https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated MAPREDUCE-7292: --- Status: Patch Available (was: Open) > Use snapshot diff to build file listing when copying to blob storage > > > Key: MAPREDUCE-7292 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7292 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Reporter: John Liu >Priority: Minor > Attachments: MAPREDUCE-7292.001.patch > > > The DistCp sync option should be extensible for copying to blob storage, > which is not a distributed filesystem. Clients of DistCp could benefit from > using the HDFS snapshot diff report to create the file listing in less time. > A valid use case is to copy new files added to HDFS to a remote blob storage. > The client ensures all new files are copied over but does not require the > destination filesystem to be a distributed filesystem or have the previous > snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7293) All pages in JHS should honor yarn.webapp.filter-entity-list-by-user
[
https://issues.apache.org/jira/browse/MAPREDUCE-7293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wanqiang Ji updated MAPREDUCE-7293:
---
Component/s: jobhistoryserver
> All pages in JHS should honor yarn.webapp.filter-entity-list-by-user
>
>
> Key: MAPREDUCE-7293
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7293
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
>
> Currently only HsJobsBlock checks for the access. If user who doesn't have
> permission to access job page is able to do it which is wrong. So we need to
> have below check in HsJobBlock,HsTasksBlock and HsTaskPage
> {code:java}
> if (isFilterAppListByUserEnabled && ugi != null && !aclsManager
> .checkAccess(ugi, JobACL.VIEW_JOB, job.getUserName(), null)) {
>
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7229) Job History Server should ignore continuing move done_intermediate files when rename failed
[
https://issues.apache.org/jira/browse/MAPREDUCE-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wanqiang Ji updated MAPREDUCE-7229:
---
Component/s: jobhistoryserver
> Job History Server should ignore continuing move done_intermediate files when
> rename failed
> ---
>
> Key: MAPREDUCE-7229
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7229
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 3.1.2
>Reporter: Morty Zhong
>Priority: Major
>
> {code:java}
> private void moveToDoneNow(final Path src, final Path target)
> throws IOException {
> LOG.info("Moving " + src.toString() + " to " + target.toString());
> try {
> intermediateDoneDirFc.rename(src, target, Options.Rename.NONE);
> } catch (FileNotFoundException e) {
> if (doneDirFc.util().exists(target)) {
> LOG.info("Source file " + src.toString() + " not found, but target "
> + "file " + target.toString() + " already exists. Move already "
> + "happened.");
> } else {
> throw e;
> }
> }
> }
> {code}
> if files in done_intermediate dir were already deleted(deleted manually), not
> move to done dir. Job history will infinitely rename these files later, and
> throw exp always.This will cause newly created files in done_intermediate dir
> can not be moved. Later, when scan done_intermediate dir,
> FileContext.listStatus(done_intermediate) will so slow, and gc problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7276) hadoop job fast fail is not working properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated MAPREDUCE-7276: --- Status: Patch Available (was: Open) > hadoop job fast fail is not working properly > - > > Key: MAPREDUCE-7276 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7276 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.9.2, 3.2.0 > Environment: hadoop 2.9.2, hive 2.3.2 and hive 0.13 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Minor > > User operation : run a hive job with HFDS disk quota specified, and enable > the fast fail feature to fast fail a job when hdfs disk quota limitation is > exceeded (mapreduce.job.dfs.storage.capacity.kill-limit-exceed=true). > Expect behaviour : the job fails fast when one task attempt fails due to > exceeding the HDFS quota limitation. > Actual behaviour : fast fail is not working at all. > Cause of the bug : TaskAttemptImpl does not handle fast fail at all in the > FAILED_FINISHING_TRANSITION event. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7276) hadoop job fast fail is not working properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji reassigned MAPREDUCE-7276: -- Assignee: Wang Yan > hadoop job fast fail is not working properly > - > > Key: MAPREDUCE-7276 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7276 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 3.2.0, 2.9.2 > Environment: hadoop 2.9.2, hive 2.3.2 and hive 0.13 >Reporter: Wang Yan >Assignee: Wang Yan >Priority: Minor > > User operation : run a hive job with HFDS disk quota specified, and enable > the fast fail feature to fast fail a job when hdfs disk quota limitation is > exceeded (mapreduce.job.dfs.storage.capacity.kill-limit-exceed=true). > Expect behaviour : the job fails fast when one task attempt fails due to > exceeding the HDFS quota limitation. > Actual behaviour : fast fail is not working at all. > Cause of the bug : TaskAttemptImpl does not handle fast fail at all in the > FAILED_FINISHING_TRANSITION event. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
