[jira] [Updated] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Jason Lowe (JIRA) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-7149:
--
Fix Version/s: 3.3.0

> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS

2018-10-11 Thread Robert Kanter (JIRA) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647088#comment-16647088
 ] 

Robert Kanter commented on MAPREDUCE-4669:
--

The 002 patch:
 * Rebased on the latest version of YARN-8448 (the 008 patch)
 * Renamed OFF, OPTIONAL, REQUIRED to NONE, LENIENT, and STRICT as per YARN-8448

> MRAM web UI does not work with HTTPS
> 
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus 
> the MRAM process cannot read the cluster keystore files to get the 
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster 
> provided) from the keystore used by AMs (which ought to be user provided).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS

2018-10-11 Thread Robert Kanter (JIRA) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated MAPREDUCE-4669:
-
Attachment: MAPREDUCE-4669.002.patch

> MRAM web UI does not work with HTTPS
> 
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus 
> the MRAM process cannot read the cluster keystore files to get the 
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster 
> provided) from the keystore used by AMs (which ought to be user provided).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647039#comment-16647039
 ] 

Hudson commented on MAPREDUCE-7149:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15181 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15181/])
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention 
(stevel: rev 735a36afd528862d7690bee2d71b2f60dd282d21)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputFormat.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/OutputFormat.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java


> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Steve Loughran (JIRA) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-7149:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

thanks, committed

> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Jason Lowe (JIRA) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646697#comment-16646697
 ] 

Jason Lowe commented on MAPREDUCE-7149:
---

Thanks for the patch!  Other than the whitespace nit, +1 lgtm.

> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646440#comment-16646440
 ] 

Hadoop QA commented on MAPREDUCE-7149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  5s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 50s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
47s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7149 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12943420/MAPREDUCE-7149-001.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux dffbdd467cc7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8a37983 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7495/artifact/out/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7495/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
U:

[jira] [Commented] (MAPREDUCE-243) Allow hadoop to run in an osgi container

2018-10-11 Thread Steve Loughran (JIRA) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646376#comment-16646376
 ] 

Steve Loughran commented on MAPREDUCE-243:
--

revisiting this. I think it's time to close as a WONTFIX, as the trend towards 
isolation is generally some linux container. But we also need to embrace java 
9+ module isolation to avoid transitive classpath hell. separate issue, covered 
elsewhere

> Allow hadoop to run in an osgi container
> 
>
> Key: MAPREDUCE-243
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-243
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: David Savage
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: classpath.patch, tasklog.patch, taskrunner.patch, 
> tasktracker.patch
>
>
> I have been running some tests getting hadoop to run within an osgi 
> environment (specifically the Newton framework) and this has uncovered a 
> number of minor bugs when mapred classes are instantiated from a different 
> start point than their main methods.
> I have created a number of patches which I'll attach which solve these 
> issues. It's possible these patches could be dealt with as separate issues 
> but all are required to resolve the osgi issue. Happy to split up if easier 
> to manage though.
> classpath.patch: this rearranges the classloader hierarchies for Task objects 
> such that a Task is able to resolve api classes in the case where the api 
> classes are no longer loaded from the system classloader.
> tasklog.patch: this ensures the log files are able to be resolved in the case 
> where the child process is launched from a different directory to the parent 
> process
> taskrunner.patch: this enables the TaskRunner to find a log dir in the case 
> where the parent jvm is not launched by the hadoop scripts, also allows for a 
> client to specify a substitute main class (which delegates to the 
> TaskTracker$Child) in this case for purposes of resolving osgi classpaths but 
> could be more general? Finally adds some extra logging in case where things 
> go wrong.
> tasktracker.patch: allow parent to pass through configuration to child 
> taskrunner (specifically in this case for purposes of passing classpath and 
> laucher to taskrunner)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Steve Loughran (JIRA) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-7149:
--
Status: Patch Available  (was: Open)

patch 001: adds the javadocs to mrv1 and v2 classes

> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Steve Loughran (JIRA) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-7149:
--
Attachment: MAPREDUCE-7149-001.patch

> javadocs for FileInputFormat and OutputFormat to mention DT collection
> --
>
> Key: MAPREDUCE-7149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-7149-001.patch
>
>
> the fact that DTs are collected for a job in 
> {{FileInputFormat.listStatus(JobConf job)}}  and 
> {{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
> or that obvious when you look @ the API.
> Add a sentence to the javadocs of the relevant methods to make clear "the job 
> you pass in is altered; anyone subclassing needs call their superclass or do 
> something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7149) javadocs for FileInputFormat and OutputFormat to mention DT collection

2018-10-11 Thread Steve Loughran (JIRA) 
Steve Loughran created MAPREDUCE-7149:
-

 Summary: javadocs for FileInputFormat and OutputFormat to mention 
DT collection
 Key: MAPREDUCE-7149
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7149
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran


the fact that DTs are collected for a job in 
{{FileInputFormat.listStatus(JobConf job)}}  and 
{{OutputFormat.checkOutputSpecs}} is not something mentioned in the javadocs, 
or that obvious when you look @ the API.

Add a sentence to the javadocs of the relevant methods to make clear "the job 
you pass in is altered; anyone subclassing needs call their superclass or do 
something similar



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org