[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated MAPREDUCE-3383: Labels: BB2015-05-RFC (was: BB2015-05-TBR) Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-RFC Attachments: MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533970#comment-14533970
]
Akira AJISAKA commented on MAPREDUCE-6079:
--
+1 pending Jenkins, thanks Tsuyoshi.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6269) improve JobConf to add option to not share Credentials between jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533958#comment-14533958 ] zhihai xu commented on MAPREDUCE-6269: -- update the patch based on the latest code. improve JobConf to add option to not share Credentials between jobs. Key: MAPREDUCE-6269 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6269 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Attachments: MAPREDUCE-6269.000.patch Improve JobConf to add constructor to avoid sharing Credentials between jobs. By default the Credentials will be shared to keep the backward compatibility. We can add a new constructor with a new parameter to decide whether to share Credentials. Some issues reported in cascading is due to corrupted credentials at https://github.com/Cascading/cascading/commit/45b33bb864172486ac43782a4d13329312d01c0e If we add this support in JobConf, it will benefit all job clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6269) improve JobConf to add option to not share Credentials between jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6269: - Attachment: MAPREDUCE-6269.000.patch improve JobConf to add option to not share Credentials between jobs. Key: MAPREDUCE-6269 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6269 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Attachments: MAPREDUCE-6269.000.patch Improve JobConf to add constructor to avoid sharing Credentials between jobs. By default the Credentials will be shared to keep the backward compatibility. We can add a new constructor with a new parameter to decide whether to share Credentials. Some issues reported in cascading is due to corrupted credentials at https://github.com/Cascading/cascading/commit/45b33bb864172486ac43782a4d13329312d01c0e If we add this support in JobConf, it will benefit all job clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6079:
-
Target Version/s: 2.8.0
Affects Version/s: 2.7.0
Labels: BB2015-05-RFC (was: BB2015-05-TBR)
Hadoop Flags: Reviewed
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6269) improve JobConf to add option to not share Credentials between jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6269: - Attachment: (was: MAPREDUCE-6269.000.patch) improve JobConf to add option to not share Credentials between jobs. Key: MAPREDUCE-6269 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6269 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Improve JobConf to add constructor to avoid sharing Credentials between jobs. By default the Credentials will be shared to keep the backward compatibility. We can add a new constructor with a new parameter to decide whether to share Credentials. Some issues reported in cascading is due to corrupted credentials at https://github.com/Cascading/cascading/commit/45b33bb864172486ac43782a4d13329312d01c0e If we add this support in JobConf, it will benefit all job clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533937#comment-14533937
]
Gera Shegalov commented on MAPREDUCE-6174:
--
Hi [~eepayne], thank you for working on the patch. We can check what checkstyle
problems a patch introduced as follows.
Run
{code}
$ mvn clean checkstyle:checkstyle-aggregate
{code}
before and after applying the patch.
Save the resulting {{target/test/checkstyle-errors.xml}} after each run. Then
we can diff both files. The diff will be concise if we get rid of line numbers.
{code}
]$ diff (cat ~/checkstyle-errors-trunk.xml | sed 's/line=\[0-9]*\//g') (cat
./target/test/checkstyle-errors.xml | sed 's/line=\[0-9]*\//g')
46768a46769,46772
file
name=/Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/IFileWrappedMapOutput.java
error column=40 severity=error message=apos;confapos; hides a
field.
source=com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck/
error column=51 severity=error message=apos;mergerapos; hides a
field.
source=com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck/
/file
46770d46773
error severity=error message=apos;newapos; have incorrect indentation
level 6, expected level should be 8.
source=com.puppycrawl.tools.checkstyle.checks.indentation.IndentationCheck/
46911d46913
error severity=error message=Line is longer than 80 characters (found
109). source=com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck/
{code}
Combine common stream code into parent class for InMemoryMapOutput and
OnDiskMapOutput.
---
Key: MAPREDUCE-6174
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 3.0.0, 2.6.0
Reporter: Eric Payne
Assignee: Eric Payne
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch,
MAPREDUCE-6174.v1.txt
Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing
similar things with regards to IFile streams.
In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are
different from 3rd-party implementations, this JIRA will make them subclass a
common class (see
https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533976#comment-14533976 ] Tsuyoshi Ozawa commented on MAPREDUCE-6284: --- +1, checking this in. Note that this feature is alpha lik [YARN's Application State Change API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_State_API] as described in the documentation. Add a 'task attempt state' to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6284: -- Summary: Add Task Attempt State API to MapReduce Application Master REST API (was: Add a 'task attempt state' to MapReduce Application Master REST API) Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6020) Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job cou
[ https://issues.apache.org/jira/browse/MAPREDUCE-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533948#comment-14533948 ] Ameya Sakhalkar commented on MAPREDUCE-6020: Looks good to me. Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress - Key: MAPREDUCE-6020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6020 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.10 Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Attachments: MAPREDUCE-6020.branch1.patch Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress. It may be a lot of JobClients to call getJobCounters in JobTracker at the same time, Current code will lock the JobTracker to block all the threads to get counter from JobInProgress. It is better to unlock the JobTracker when get counter from JobInProgress(job.getCounters(counters)). So all the theads can run parallel when access its own job counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java and TaskLogServlet.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Attachment: 0001-MAPREDUCE-2631.patch
Rebasing patch against trunk. Patch still looks valid.
Activity done as part of bugbash0515.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java and
TaskLogServlet.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Ravi Teja Ch N V
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=TaskLogServlet.java|borderStyle=solid}
private void printTaskLog(HttpServletResponse response,
OutputStream out, TaskAttemptID taskId,
long start, long end, boolean plainText,
TaskLog.LogName filter, boolean isCleanup)
throws IOException {
if (!plainText) {
out.write((brbu + filter + logs/u/bbr\n +
pre\n).getBytes());
}
try {
InputStream taskLogReader =
new TaskLog.Reader(taskId, filter, start, end, isCleanup);
byte[] b = new byte[65536];
int result;
while (true) {
result = taskLogReader.read(b);
if (result 0) {
if (plainText) {
out.write(b, 0, result);
} else {
HtmlQuoting.quoteHtmlChars(out, b, 0, result);
}
} else {
break;
}
}
taskLogReader.close();
{code}
In the above code, if any exception thrown while reading
(taskLogReader.read(b)), taskLogReader will not be closed.
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534178#comment-14534178
]
Akira AJISAKA commented on MAPREDUCE-6079:
--
Copied from
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5666/artifact/patchprocess/commentfile
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 36s | The applied patch generated 1
new checkstyle issues (total was 107, now 107). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 59s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 47m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731370/MAPREDUCE-6079.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 26da2dd |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5666/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5666/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5666/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5666/console |
This message was automatically generated.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6079:
-
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk and branch-2. Thanks [~ozawa] for the contribution.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6284: -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~ryu_kobayashi] for your contribution! Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534056#comment-14534056 ] Hudson commented on MAPREDUCE-6284: --- FAILURE: Integrated in Hadoop-trunk-Commit #7771 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7771/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/CHANGES.txt Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java and TaskLogServlet.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Assignee: Sunil G (was: Ravi Teja Ch N V)
Labels: BB2015-05-RFC (was: )
Status: Patch Available (was: Open)
Submitting the change to kick jenkins.
[~raviteja] pls reassign if you are interested still.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java and
TaskLogServlet.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Labels: BB2015-05-RFC
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=TaskLogServlet.java|borderStyle=solid}
private void printTaskLog(HttpServletResponse response,
OutputStream out, TaskAttemptID taskId,
long start, long end, boolean plainText,
TaskLog.LogName filter, boolean isCleanup)
throws IOException {
if (!plainText) {
out.write((brbu + filter + logs/u/bbr\n +
pre\n).getBytes());
}
try {
InputStream taskLogReader =
new TaskLog.Reader(taskId, filter, start, end, isCleanup);
byte[] b = new byte[65536];
int result;
while (true) {
result = taskLogReader.read(b);
if (result 0) {
if (plainText) {
out.write(b, 0, result);
} else {
HtmlQuoting.quoteHtmlChars(out, b, 0, result);
}
} else {
break;
}
}
taskLogReader.close();
{code}
In the above code, if any exception thrown while reading
(taskLogReader.read(b)), taskLogReader will not be closed.
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned MAPREDUCE-2632: -- Assignee: Sunil G (was: Ravi Teja Ch N V) Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Labels: BB2015-05-TBR Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Summary: Potential resource leaks in BinaryProtocol$TeeOutputStream.java
(was: Potential resource leaks in BinaryProtocol$TeeOutputStream.java and
TaskLogServlet.java)
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Labels: BB2015-05-RFC
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java and TaskLogServlet.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Description:
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
was:
{code:title=TaskLogServlet.java|borderStyle=solid}
private void printTaskLog(HttpServletResponse response,
OutputStream out, TaskAttemptID taskId,
long start, long end, boolean plainText,
TaskLog.LogName filter, boolean isCleanup)
throws IOException {
if (!plainText) {
out.write((brbu + filter + logs/u/bbr\n +
pre\n).getBytes());
}
try {
InputStream taskLogReader =
new TaskLog.Reader(taskId, filter, start, end, isCleanup);
byte[] b = new byte[65536];
int result;
while (true) {
result = taskLogReader.read(b);
if (result 0) {
if (plainText) {
out.write(b, 0, result);
} else {
HtmlQuoting.quoteHtmlChars(out, b, 0, result);
}
} else {
break;
}
}
taskLogReader.close();
{code}
In the above code, if any exception thrown while reading
(taskLogReader.read(b)), taskLogReader will not be closed.
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java and
TaskLogServlet.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Labels: BB2015-05-RFC
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4070) JobHistoryServer creates /tmp directory with restrictive permissions if the directory doesn't already exist.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534166#comment-14534166
]
Ameya Sakhalkar commented on MAPREDUCE-4070:
Tried to reproduce with version 2.5.0. Deleted /tmp and restarted
HistoryServer. It created /tmp with permissions 770:
drwxrwx--- - hadoop supergroup 0 2015-05-08 14:17 /tmp
JobHistoryServer creates /tmp directory with restrictive permissions if the
directory doesn't already exist.
Key: MAPREDUCE-4070
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4070
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-4070.patch
Starting up the MapReduce JobhHistoryServer service after a clean install
appears to automatically create the /tmp directory on HDFS. However, it is
created with 750 permission.
Attempting to run MR jobs by other users results in the following permissions
exception:
{code}
org.apache.hadoop.security.AccessControlException: Permission denied:
user=cloudera, access=EXECUTE, inode=/tmp:yarn:supergroup:drwxr-x---
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)
..
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534179#comment-14534179
]
Akira AJISAKA commented on MAPREDUCE-6079:
--
bq. -1 checkstyle
This is because the file length of JobImpl.java is longer than 2,000 lines.
Committing this.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6079:
-
Labels: (was: BB2015-05-RFC)
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6079:
-
Summary: Rename JobImpl#username to reporterUserName (was: Renaming
JobImpl#username to reporterUserName)
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534204#comment-14534204
]
Hudson commented on MAPREDUCE-6079:
---
FAILURE: Integrated in Hadoop-trunk-Commit #7772 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7772/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* hadoop-mapreduce-project/CHANGES.txt
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3383: - Labels: BB2015-05-TBR (was: BB2015-05-RFC) Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-TBR Attachments: MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6020) Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job count
[ https://issues.apache.org/jira/browse/MAPREDUCE-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6020: - Attachment: MAPREDUCE-6020.branch1.patch Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress - Key: MAPREDUCE-6020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6020 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.10 Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Attachments: MAPREDUCE-6020.branch1.patch Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress. It may be a lot of JobClients to call getJobCounters in JobTracker at the same time, Current code will lock the JobTracker to block all the threads to get counter from JobInProgress. It is better to unlock the JobTracker when get counter from JobInProgress(job.getCounters(counters)). So all the theads can run parallel when access its own job counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6020) Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job count
[ https://issues.apache.org/jira/browse/MAPREDUCE-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6020: - Attachment: (was: MAPREDUCE-6020.branch1.patch) Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress - Key: MAPREDUCE-6020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6020 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.10 Reporter: zhihai xu Assignee: zhihai xu Labels: BB2015-05-TBR Attachments: MAPREDUCE-6020.branch1.patch Too many threads blocking on the global JobTracker lock from getJobCounters, optimize getJobCounters to release global JobTracker lock before access the per job counter in JobInProgress. It may be a lot of JobClients to call getJobCounters in JobTracker at the same time, Current code will lock the JobTracker to block all the threads to get counter from JobInProgress. It is better to unlock the JobTracker when get counter from JobInProgress(job.getCounters(counters)). So all the theads can run parallel when access its own job counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Attachment: MAPREDUCE-6342-branch-2.patch Updated patch for branch-2 Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated MAPREDUCE-3383: Attachment: MAPREDUCE-3383.patch re submitting the patch after correcting the format Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-TBR Attachments: MAPREDUCE-3383.patch, MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-2632: --- Attachment: 0001-MAPREDUCE-2632.patch Seems patch is not as per latest trunk. i rebased and applied. Patch still is valid. Doing as part of BugBash Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Ravi Teja Ch N V Labels: BB2015-05-TBR Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6284: -- Issue Type: New Feature (was: Improvement) Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3383:
-
Target Version/s: 2.8.0
Status: Open (was: Patch Available)
Thanks [~decster] for the patch and [~mohdshahidkhan] for review.
Changes looks good to me except there is a whitespace warning with the patch,
can you have look into this?
{code:xml}
stdin:10: trailing whitespace.
comparator, keyClass, valueClass,
warning: 1 line adds whitespace errors.
{code}
Duplicate job.getOutputValueGroupingComparator() in ReduceTask
--
Key: MAPREDUCE-3383
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Binglin Chang
Assignee: Binglin Chang
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-3383.patch
This is probably just a small error by mistake.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6273) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
[
https://issues.apache.org/jira/browse/MAPREDUCE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534190#comment-14534190
]
Tsuyoshi Ozawa commented on MAPREDUCE-6273:
---
[~zxu], could you add a test case to TestHistoryFileManager?
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
Key: MAPREDUCE-6273
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6273
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6273.000.patch
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state,
I saw the following error message:
{code}
2015-02-17 19:13:45,198 ERROR
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to
move a job to done
java.io.FileNotFoundException: File does not exist:
/user/history/done_intermediate/agd_laci-sluice/job_1423740288390_1884.summary
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1878)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1771)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:85)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1181)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:230)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:318)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:59)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:621)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:789)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:785)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:785)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getJobSummary(HistoryFileManager.java:953)
at
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-3383: -- Status: Patch Available (was: Open) Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-TBR Attachments: MAPREDUCE-3383.patch, MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-5981: Labels: BB2015-05-RFC (was: BB2015-05-TBR) Target Version/s: 2.8.0 Affects Version/s: 2.7.0 Status: Patch Available (was: Open) Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Labels: BB2015-05-RFC Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-5981: Attachment: MAPREDUCE-5981.02.patch Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Labels: BB2015-05-TBR Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4127) Resource manager UI does not show the Job Priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K resolved MAPREDUCE-4127. -- Resolution: Duplicate It will be handled as part of YARN-1963. Resource manager UI does not show the Job Priority -- Key: MAPREDUCE-4127 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4127 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Nishan Shetty In RM UI the priority of job is not displayed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-5981: Description: Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); was: Following map reduce logs can be changed to DEBUG log level. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Labels: BB2015-05-TBR Attachments: MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6342: - Fix Version/s: 2.8.0 Labels: (was: BB2015-05-RFC) Hadoop Flags: Reviewed +1, patch looks good to me, will commit it shortly. Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-5981: Status: Open (was: Patch Available) Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Varun Saxena Assignee: Varun Saxena Labels: BB2015-05-TBR Attachments: MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2393) No total min share limitation of all pools
[ https://issues.apache.org/jira/browse/MAPREDUCE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534286#comment-14534286 ] Mohammad Shahid Khan commented on MAPREDUCE-2393: - MRV2 architecture is now changed and as per current scenario looks like invalid issue. No total min share limitation of all pools -- Key: MAPREDUCE-2393 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2393 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.21.0 Reporter: Denny Ye Labels: BB2015-05-TBR, fair, scheduler Attachments: MAPREDUCE-2393.patch hi, there is no limitation about min share of all pools with cluster total shares. User can define arbitrary amount of min share for each pool. It has such description in fair scheduler design document, but no regular code. It may critical for slot distribution. One pool can hold all cluster slots to meet it's min share that greater than cluster total slots very much. If that case has happened, we should scaled down proportionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2393) No total min share limitation of all pools
[ https://issues.apache.org/jira/browse/MAPREDUCE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated MAPREDUCE-2393: Labels: BB2015-05-RFC (was: BB2015-05-TBR fair scheduler) No total min share limitation of all pools -- Key: MAPREDUCE-2393 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2393 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.21.0 Reporter: Denny Ye Labels: BB2015-05-RFC Attachments: MAPREDUCE-2393.patch hi, there is no limitation about min share of all pools with cluster total shares. User can define arbitrary amount of min share for each pool. It has such description in fair scheduler design document, but no regular code. It may critical for slot distribution. One pool can hold all cluster slots to meet it's min share that greater than cluster total slots very much. If that case has happened, we should scaled down proportionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6342: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~rohithsharma]. Committed to trunk and branch-2. Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534250#comment-14534250 ] Hudson commented on MAPREDUCE-6342: --- FAILURE: Integrated in Hadoop-trunk-Commit #7773 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7773/]) MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. (devaraj: rev 2401ff76bea603b2c4e59338d5954ef60d6c4005) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2393) No total min share limitation of all pools
[ https://issues.apache.org/jira/browse/MAPREDUCE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated MAPREDUCE-2393: Resolution: Won't Fix Status: Resolved (was: Patch Available) as per current scenario seems like invalid. Please reopen if issue yet exist. No total min share limitation of all pools -- Key: MAPREDUCE-2393 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2393 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.21.0 Reporter: Denny Ye Labels: BB2015-05-RFC Attachments: MAPREDUCE-2393.patch hi, there is no limitation about min share of all pools with cluster total shares. User can define arbitrary amount of min share for each pool. It has such description in fair scheduler design document, but no regular code. It may critical for slot distribution. One pool can hold all cluster slots to meet it's min share that greater than cluster total slots very much. If that case has happened, we should scaled down proportionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534540#comment-14534540
]
Hudson commented on MAPREDUCE-6079:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534533#comment-14534533 ] Hudson commented on MAPREDUCE-6284: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534505#comment-14534505
]
Hudson commented on MAPREDUCE-6079:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/921/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* hadoop-mapreduce-project/CHANGES.txt
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3383: - Attachment: MAPREDUCE-3383-1.patch Updating the patch with white space fix. Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-RFC Attachments: MAPREDUCE-3383-1.patch, MAPREDUCE-3383.patch, MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3383: - Status: Patch Available (was: Open) Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-RFC Attachments: MAPREDUCE-3383-1.patch, MAPREDUCE-3383.patch, MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534483#comment-14534483 ] Hudson commented on MAPREDUCE-6342: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/921/]) MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. (devaraj: rev 2401ff76bea603b2c4e59338d5954ef60d6c4005) * hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * hadoop-mapreduce-project/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534475#comment-14534475 ] Hudson commented on MAPREDUCE-6279: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/921/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534498#comment-14534498 ] Hudson commented on MAPREDUCE-6356: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/921/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534496#comment-14534496 ] Hudson commented on MAPREDUCE-6284: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/921/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[
https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534615#comment-14534615
]
Hadoop QA commented on MAPREDUCE-5981:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 32s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 48s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 49s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731427/MAPREDUCE-5981.02.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9757864 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5673/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5673/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5673/console |
This message was automatically generated.
Log levels of certain MR logs can be changed to DEBUG
-
Key: MAPREDUCE-5981
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Varun Saxena
Assignee: Varun Saxena
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch
Following map reduce logs can be changed to DEBUG log level as they appear
too many times in the log file and are not that important for debugging.
1. In
org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java :
313), the second log is not required to be at info level. This can be moved
to debug as a warn log is anyways printed if verifyReply fails.
SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);
LOG.info(for url=+msgToEncode+ sent hash and received reply);
2. Thread related info need not be printed in logs at INFO level. Below 2
logs can be moved to DEBUG
a) In
org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java
: 381), below log can be changed to DEBUG
LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() +
to + Thread.currentThread().getName());
b) In
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java
: 411), below log can be changed to DEBUG
LOG.info(assigned + includedMaps + of + totalSize + to +
host + to + Thread.currentThread().getName());
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3383: - Status: Open (was: Patch Available) whitespace error appears with the latest patch as well. Duplicate job.getOutputValueGroupingComparator() in ReduceTask -- Key: MAPREDUCE-3383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Binglin Chang Assignee: Binglin Chang Labels: BB2015-05-RFC Attachments: MAPREDUCE-3383.patch, MAPREDUCE-3383.patch This is probably just a small error by mistake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534627#comment-14534627
]
Devaraj K commented on MAPREDUCE-2631:
--
Thanks [~sunilg] for the updated patch.
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5670/console
{code:xml}
-1 overall
| Vote | Subsystem | Runtime | Comment
| 0 | pre-patch | 14m 46s | Pre-patch trunk compilation is
| | || healthy.
| +1 |@author | 0m 0s | The patch does not contain any
| | || @author tags.
| +1 | tests included | 0m 0s | The patch appears to include 2 new
| | || or modified test files.
| +1 | javac | 7m 39s| There were no new javac warning
| | || messages.
| +1 |javadoc | 9m 51s| There were no new javadoc warning
| | || messages.
| +1 | release audit | 0m 21s| The applied patch does not increase
| | || the total number of release audit
| | || warnings.
| +1 | checkstyle | 1m 16s| There were no new checkstyle
| | || issues.
| +1 | whitespace | 0m 0s | The patch has no lines that end in
| | || whitespace.
| +1 |install | 1m 33s| mvn install still works.
| +1 |eclipse:eclipse | 0m 32s| The patch built with
| | || eclipse:eclipse.
| +1 | findbugs | 1m 56s| The patch does not introduce any
| | || new Findbugs (version 2.0.3)
| | || warnings.
| +1 |mapreduce tests | 1m 36s| Tests passed in
| | || hadoop-mapreduce-client-core.
| -1 |mapreduce tests | 97m 32s | Tests failed in
| | || hadoop-mapreduce-client-jobclient.
| | | 137m 6s |
Reason | Tests
Failed unit tests | hadoop.mapreduce.security.ssl.TestEncryptedShuffle
| hadoop.mapreduce.security.TestBinaryTokenFile
| hadoop.mapreduce.TestMapReduceLazyOutput
| hadoop.mapred.pipes.TestPipeApplication
| hadoop.mapreduce.TestMRJobClient
{code}
Can you have a look into these failed tests?
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534522#comment-14534522 ] Hudson commented on MAPREDUCE-6342: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/]) MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. (devaraj: rev 2401ff76bea603b2c4e59338d5954ef60d6c4005) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534514#comment-14534514 ] Hudson commented on MAPREDUCE-6279: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #190 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/190/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534669#comment-14534669 ] Hudson commented on MAPREDUCE-6279: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534677#comment-14534677 ] Hudson commented on MAPREDUCE-6342: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/]) MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. (devaraj: rev 2401ff76bea603b2c4e59338d5954ef60d6c4005) * hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml * hadoop-mapreduce-project/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/CHANGES.txt Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534691#comment-14534691 ] Hudson commented on MAPREDUCE-6284: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/CHANGES.txt Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534699#comment-14534699
]
Hudson commented on MAPREDUCE-6079:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* hadoop-mapreduce-project/CHANGES.txt
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534735#comment-14534735 ] Ray Chiang commented on MAPREDUCE-6192: --- Thanks for the review and the commit! Create unit test to automatically compare MR related classes and mapred-default.xml --- Key: MAPREDUCE-6192 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Fix For: 2.8.0 Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch, MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch, MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch, MAPREDUCE-6192.branch-2.007.patch Create a unit test that will automatically compare the fields in the various MapReduce related classes and mapred-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-695) MiniMRCluster while shutting down should not wait for currently running jobs to finish
[
https://issues.apache.org/jira/browse/MAPREDUCE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-695:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
MiniMRCluster is wrapper of MiniMRYarnCluster now and {{waitTaskTrackers}} is
not used from anywhere.
{code}
private void waitTaskTrackers() {
throw new UnsupportedOperationException();
}
{code}
I'm closing this as won't fix.
MiniMRCluster while shutting down should not wait for currently running jobs
to finish
--
Key: MAPREDUCE-695
URL: https://issues.apache.org/jira/browse/MAPREDUCE-695
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.0.3
Reporter: Sreekanth Ramakrishnan
Priority: Minor
Attachments: mapreduce-695.patch
Currently in {{org.apache.hadoop.mapred.MiniMRCluster.shutdown()}} we do a
{{waitTaskTrackers()}} which can cause {{MiniMRCluster}} to hang indefinitely
when used in conjunction with Controlled jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-2632: Attachment: mr-2632-3.patch Fair point, Vinod. Here is a patch that updates the javadoc to capture this. Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch, mr-2632-2.patch, mr-2632-3.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534943#comment-14534943 ] Hudson commented on MAPREDUCE-5981: --- FAILURE: Integrated in Hadoop-trunk-Commit #7776 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7776/]) MAPREDUCE-5981. Log levels of certain MR logs can be changed to DEBUG. (devaraj: rev dc2b2ae31f2eb6dae324c2e14ed7660ce605a89b) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.8.0 Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated MAPREDUCE-6246: -- Labels: (was: BB2015-05-RFC.) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 - Key: MAPREDUCE-6246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.4.1 Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x Platform: xSeries, pSeries Browser: Firefox, IE Security Settings: No Security, Flat file, LDAP, PAM File System: HDFS, GPFS FPO Reporter: ramtin Assignee: ramtin Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.patch Original Estimate: 24h Remaining Estimate: 24h DBoutputformat is used for writing output of mapreduce jobs to the database and when used with db2 jdbc drivers it fails with following error com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT, DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127) In DBOutputFormat class there is constructQuery method that generates INSERT INTO statement with semicolon(;) at the end. Semicolon is ANSI SQL-92 standard character for a statement terminator but this feature is disabled(OFF) as a default settings in IBM DB2. Although by using -t we can turn it ON for db2. (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2). But there are some products that already built on top of this default setting (OFF) so by turning ON this feature make them error prone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5465) Container killed before hprof dumps profile.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated MAPREDUCE-5465: --- Labels: BB2015-05-RFC (was: BB2015-05-TBR) Container killed before hprof dumps profile.out --- Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Labels: BB2015-05-RFC Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-1290) DBOutputFormat does not support rewriteBatchedStatements when using MySQL jdbc drivers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-1290: -- Labels: DBOutoutFormat patch (was: BB2015-05-TBR DBOutoutFormat patch) DBOutputFormat does not support rewriteBatchedStatements when using MySQL jdbc drivers -- Key: MAPREDUCE-1290 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1290 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: Joe Crobak Labels: DBOutoutFormat, patch Attachments: MAPREDUCE-1290.patch, MapReduce-1290-trunk.patch The DBOutputFormat adds a semi-colon to the end of the INSERT statement that it uses to save fields to the database. Semicolons are typically used in command line programs but are not needed when using the JDBC API. In this case, the stray semi-colon breaks rewriteBatchedStatement support. See: http://forums.mysql.com/read.php?39,271526,271526#msg-271526 for an example. In my use case, rewriteBatchedStatement is very useful because it increases the speed of inserts and reduces memory consumption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-5981: - Hadoop Flags: Reviewed +1, latest patch looks good to me, will commit it shortly. Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Labels: BB2015-05-RFC Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3385) Add warning message for the overflow in reduce() of org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-3385: -- Labels: (was: BB2015-05-TBR) Add warning message for the overflow in reduce() of org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer Key: MAPREDUCE-3385 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3385 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Minor Attachments: MAPREDUCE-3385.patch When we call the function reduce() of IntSumReducer,the result may overflow. We should send a warning message to users if overflow occurs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-2632: Attachment: mr-2632-2.patch Here is a patch that fixes the nit. And, I verified the test passes locally. Don't see the need for kicking Jenkins off on this one, given the high load. Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch, mr-2632-2.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-2632: Target Version/s: 3.0.0 Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch, mr-2632-2.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534900#comment-14534900 ] Hudson commented on MAPREDUCE-6279: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534932#comment-14534932
]
Hudson commented on MAPREDUCE-6079:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* hadoop-mapreduce-project/CHANGES.txt
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534993#comment-14534993 ] Hudson commented on MAPREDUCE-6284: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/189/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6174: -- Labels: (was: BB2015-05-TBR) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-2632: Component/s: (was: tasktracker) Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5663) Add an interface to Input/Ouput Formats to obtain delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534851#comment-14534851 ] Roman Shaposhnik commented on MAPREDUCE-5663: - [~sseth] I'm looking at it as part of bugbash (and also b/c krb is something I'm pretty familiar with). Given the discussion so far, I'm not quite sure where this stands. The last comment from [~tucu00] makes a lot of sense to me, but obviously I'm looking at this for the first time. Add an interface to Input/Ouput Formats to obtain delegation tokens --- Key: MAPREDUCE-5663 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5663 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Siddharth Seth Assignee: Michael Weng Labels: BB2015-05-TBR Attachments: MAPREDUCE-5663.4.txt, MAPREDUCE-5663.5.txt, MAPREDUCE-5663.6.txt, MAPREDUCE-5663.patch.txt, MAPREDUCE-5663.patch.txt2, MAPREDUCE-5663.patch.txt3 Currently, delegation tokens are obtained as part of the getSplits / checkOutputSpecs calls to the InputFormat / OutputFormat respectively. This works as long as the splits are generated on a node with kerberos credentials. For split generation elsewhere (AM for example), an explicit interface is required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534862#comment-14534862 ] Karthik Kambatla commented on MAPREDUCE-2632: - For legit uses of partitioner, this shouldn't be incompatible. However, there *could* be folks out there who rely on the getPartition being called for some side-effects. Since this is an optimization and we are in no hurry to get this in, I plan to check this into trunk. The patch itself looks okay, but for one nit in TestMapFileOutputFormat: the test defines setGetPartitionCalled but never calls it. Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3385) Add warning message for the overflow in reduce() of org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-3385: -- Labels: BB2015-05-RFC (was: ) Add warning message for the overflow in reduce() of org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer Key: MAPREDUCE-3385 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3385 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-3385.patch When we call the function reduce() of IntSumReducer,the result may overflow. We should send a warning message to users if overflow occurs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2340) optimize JobInProgress.initTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated MAPREDUCE-2340: --- Attachment: MAPREDUCE-2340-branch-1.4.patch patch rebased to current branch-1. I tried looking for an equivalent operation in trunk but couldn't find anything. optimize JobInProgress.initTasks() -- Key: MAPREDUCE-2340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2340 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.1, 0.21.0 Reporter: Kang Xiao Labels: BB2015-05-TBR, critical-0.22.0 Attachments: MAPREDUCE-2340-branch-1.4.patch, MAPREDUCE-2340.patch, MAPREDUCE-2340.patch, MAPREDUCE-2340.r1.diff JobTracker's hostnameToNodeMap cache can speed up JobInProgress.initTasks() and JobInProgress.createCache() significantly. A test for 1 job with 10 maps on a 2400 cluster shows nearly 10 and 50 times speed up for initTasks() and createCache(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-695) MiniMRCluster while shutting down should not wait for currently running jobs to finish
[
https://issues.apache.org/jira/browse/MAPREDUCE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-695:
---
Labels: (was: BB2015-05-TBR)
MiniMRCluster while shutting down should not wait for currently running jobs
to finish
--
Key: MAPREDUCE-695
URL: https://issues.apache.org/jira/browse/MAPREDUCE-695
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.0.3
Reporter: Sreekanth Ramakrishnan
Priority: Minor
Attachments: mapreduce-695.patch
Currently in {{org.apache.hadoop.mapred.MiniMRCluster.shutdown()}} we do a
{{waitTaskTrackers()}} which can cause {{MiniMRCluster}} to hang indefinitely
when used in conjunction with Controlled jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534924#comment-14534924 ] Hudson commented on MAPREDUCE-6284: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5248) Let NNBenchWithoutMR specify the replication factor for its test
[ https://issues.apache.org/jira/browse/MAPREDUCE-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-5248: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) Let NNBenchWithoutMR specify the replication factor for its test Key: MAPREDUCE-5248 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5248 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, test Affects Versions: 3.0.0 Reporter: Erik Paulson Assignee: Erik Paulson Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-5248.patch, MAPREDUCE-5248.txt Original Estimate: 1h Remaining Estimate: 1h The NNBenchWithoutMR test creates files with a replicationFactorPerFile hard-coded to 1. It'd be nice to be able to specify that on the commandline. Also, it'd be great if MAPREDUCE-4750 was merged along with this fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-2632: Labels: (was: BB2015-05-RFC) Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3383) Duplicate job.getOutputValueGroupingComparator() in ReduceTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14535064#comment-14535064
]
Hadoop QA commented on MAPREDUCE-3383:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 51s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 45s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 50s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 17s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 37s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 43s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731480/MAPREDUCE-3383-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f4ebbc6 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5675/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5675/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5675/console |
This message was automatically generated.
Duplicate job.getOutputValueGroupingComparator() in ReduceTask
--
Key: MAPREDUCE-3383
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3383
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Binglin Chang
Assignee: Binglin Chang
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-3383-1.patch, MAPREDUCE-3383.patch,
MAPREDUCE-3383.patch
This is probably just a small error by mistake.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated MAPREDUCE-6246: -- Labels: BB2015-05-RFC (was: ) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 - Key: MAPREDUCE-6246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.4.1 Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x Platform: xSeries, pSeries Browser: Firefox, IE Security Settings: No Security, Flat file, LDAP, PAM File System: HDFS, GPFS FPO Reporter: ramtin Assignee: ramtin Labels: BB2015-05-RFC Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.patch Original Estimate: 24h Remaining Estimate: 24h DBoutputformat is used for writing output of mapreduce jobs to the database and when used with db2 jdbc drivers it fails with following error com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT, DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127) In DBOutputFormat class there is constructQuery method that generates INSERT INTO statement with semicolon(;) at the end. Semicolon is ANSI SQL-92 standard character for a statement terminator but this feature is disabled(OFF) as a default settings in IBM DB2. Although by using -t we can turn it ON for db2. (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2). But there are some products that already built on top of this default setting (OFF) so by turning ON this feature make them error prone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534762#comment-14534762
]
Hadoop QA commented on MAPREDUCE-2631:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 43s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 20s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 0s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 49s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 62m 4s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 101m 37s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestMiniMRClientCluster |
| | hadoop.mapred.TestIFileStreams |
| | hadoop.mapred.TestJobName |
| | hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| | hadoop.mapred.TestLazyOutput |
| Timed out tests | org.apache.hadoop.mapred.TestJobCounters |
| | org.apache.hadoop.mapred.TestMiniMRClasspath |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731401/0001-MAPREDUCE-2631.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 241a72a |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5674/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5674/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5674/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5674/console |
This message was automatically generated.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-5981: - Labels: (was: BB2015-05-RFC) Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.8.0 Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add Task Attempt State API to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534792#comment-14534792 ] Hudson commented on MAPREDUCE-6284: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/]) MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. (ozawa: rev d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptState.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredAppMasterRest.md * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java Add Task Attempt State API to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Fix For: 2.8.0 Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534773#comment-14534773 ] Hudson commented on MAPREDUCE-6279: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534781#comment-14534781 ] Hudson commented on MAPREDUCE-6342: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/]) MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. (devaraj: rev 2401ff76bea603b2c4e59338d5954ef60d6c4005) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * hadoop-mapreduce-project/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5981) Log levels of certain MR logs can be changed to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-5981: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~varun_saxena]. Committed to trunk and branch-2 Log levels of certain MR logs can be changed to DEBUG - Key: MAPREDUCE-5981 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5981 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Varun Saxena Assignee: Varun Saxena Fix For: 2.8.0 Attachments: MAPREDUCE-5981.02.patch, MAPREDUCE-5981.patch Following map reduce logs can be changed to DEBUG log level as they appear too many times in the log file and are not that important for debugging. 1. In org.apache.hadoop.mapreduce.task.reduce.Fetcher#copyFromHost(Fetcher.java : 313), the second log is not required to be at info level. This can be moved to debug as a warn log is anyways printed if verifyReply fails. SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey); LOG.info(for url=+msgToEncode+ sent hash and received reply); 2. Thread related info need not be printed in logs at INFO level. Below 2 logs can be moved to DEBUG a) In org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl#getHost(ShuffleSchedulerImpl.java : 381), below log can be changed to DEBUG LOG.info(Assigning + host + with + host.getNumKnownMapOutputs() + to + Thread.currentThread().getName()); b) In org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.getMapsForHost(ShuffleSchedulerImpl.java : 411), below log can be changed to DEBUG LOG.info(assigned + includedMaps + of + totalSize + to + host + to + Thread.currentThread().getName()); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Rename JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534799#comment-14534799
]
Hudson commented on MAPREDUCE-6079:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #179 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/179/])
MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by
Tsuyoshi Ozawa. (aajisaka: rev ecfa05227489724f12775b94a9f9184b66e865af)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* hadoop-mapreduce-project/CHANGES.txt
Rename JobImpl#username to reporterUserName
---
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.7.0
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Fix For: 2.8.0
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated MAPREDUCE-6246: -- Labels: BB2015-05-RFC. (was: BB2015-05-TBR DB2 mapreduce) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 - Key: MAPREDUCE-6246 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6246 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.4.1 Environment: OS: RHEL 5.x, RHEL 6.x, SLES 11.x Platform: xSeries, pSeries Browser: Firefox, IE Security Settings: No Security, Flat file, LDAP, PAM File System: HDFS, GPFS FPO Reporter: ramtin Assignee: ramtin Labels: BB2015-05-RFC. Attachments: MAPREDUCE-6246.002.patch, MAPREDUCE-6246.patch Original Estimate: 24h Remaining Estimate: 24h DBoutputformat is used for writing output of mapreduce jobs to the database and when used with db2 jdbc drivers it fails with following error com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=;;,COUNT) VALUES (?,?);END-OF-STATEMENT, DRIVER=4.16.53 at com.ibm.db2.jcc.am.fd.a(fd.java:739) at com.ibm.db2.jcc.am.fd.a(fd.java:60) at com.ibm.db2.jcc.am.fd.a(fd.java:127) In DBOutputFormat class there is constructQuery method that generates INSERT INTO statement with semicolon(;) at the end. Semicolon is ANSI SQL-92 standard character for a statement terminator but this feature is disabled(OFF) as a default settings in IBM DB2. Although by using -t we can turn it ON for db2. (http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0010410.html?cp=SSEPGG_9.7.0%2F3-6-2-0-2). But there are some products that already built on top of this default setting (OFF) so by turning ON this feature make them error prone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-2631:
Attachment: MAPREDUCE-2631.02.patch
Thanks for the patch Sunil. In BinaryProtocol.close(), if out.close() throws an
exception, file will never be set to null right?
Uploading a new patch which does this:
{code}
public void close() throws IOException {
try {
flush();
} finally {
IOUtils.closeStream(file);
file = null;
IOUtils.closeStream(file);
out = null;
}
}
{code}
Similarly for IFileOutputStream . Could you please review it?
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, MAPREDUCE-2631.02.patch,
MAPREDUCE-2631.1.patch, MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch,
MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534870#comment-14534870 ] Karthik Kambatla commented on MAPREDUCE-2632: - Just realized, Jenkins didn't run before the nit fix. Never mind. Will wait for Jenkins to say it is okay to commit. Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch, mr-2632-2.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534872#comment-14534872 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2632: We should change Partitioner's javadoc to indicate this. Avoid calling the partitioner when the numReduceTasks is 1. --- Key: MAPREDUCE-2632 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Teja Ch N V Assignee: Sunil G Attachments: 0001-MAPREDUCE-2632.patch, MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch, mr-2632-2.patch We can avoid the call to the partitioner when the number of reducers is 1.This will avoid the unnecessary computations by the partitioner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
