[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6377: -- Fix Version/s: (was: 2.8.0) 2.7.3 2.6.3 Thanks, [~zxu]! I committed this to branch-2.7 and branch-2.6 as well. > JHS sorting on state column not working in webUi > > > Key: MAPREDUCE-6377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.0 > Environment: 2 NM, JHS >Reporter: Bibin A Chundatt >Assignee: zhihai xu >Priority: Minor > Fix For: 2.6.3, 2.7.3 > > Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, > state_sorted1.pdf, state_sorted2.pdf > > > Steps to reproduce > > 1. Install and setup HA cluster with JHS > 2.Create state in in JHS where few jobs are killed and Success > Check sorting State in JHS WebUI > Actual > = > Sorting on state column not working in JHS > Expected > == > Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011879#comment-15011879
]
Jason Lowe commented on MAPREDUCE-6550:
---
Thanks for the patch, Robert!
I'm a bit worried about having the working directory have wide-open
permissions. Literally anyone on the cluster can go in and start rearranging
the contents of that directory, as they have write permissions to it. I think
we need to at least set the sticky bit on it so it's like /tmp. Users can
create their own stuff but they can't fiddle with other users' stuff.
Do we really want to only support the proxy user setup? Wondering if there's a
scenario where admins want to aggregate the logs to save on the namespace but
don't want to allow the proxy behaviors. As long as they're OK with fetching
the logs via the log server and not via direct HDFS access by the user, it
would still work despite the logs being owned by the HDFS user.
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6550.001.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DistributedContainerExecutor, this means that the job
> will actually run as the Yarn user, so the resulting har files are owned by
> the Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011905#comment-15011905
]
Jason Lowe commented on MAPREDUCE-5870:
---
It would be nice to cleanup most of the checkstyle nits, as there are a lot of
switch statements without default clauses that were added. I realize
semantically we're covered since the fall-through case returns some default
value or throws, but we can trivially remove the warning by having an explicit
default clause that explicitly breaks or move the fallthrough code into that
default case.
Other than that latest patch looks good to me.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010725#comment-15010725
]
Hadoop QA commented on MAPREDUCE-6542:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 55s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 2s
{color} | {color:red} Patch generated 8 new checkstyle issues in root (total
was 299, now 306). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 13s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 55s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 100m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed
[jira] [Created] (MAPREDUCE-6551) Dynamic adjust mapTaskAttempt memory size
Lin Yiqun created MAPREDUCE-6551:
Summary: Dynamic adjust mapTaskAttempt memory size
Key: MAPREDUCE-6551
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6551
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 2.7.1
Reporter: Lin Yiqun
Assignee: Lin Yiqun
I found a scenario that the map tasks cost so much resource of cluster.This
scenario will be happened that if there are many small file blokcs (even some
are not reach 1M),and this will lead to many map task to read.And in gengeral,a
map task attempt will use the default config {{MRJobConfig#MAP_MEMORY_MB}} to
set its resourceCapcity's memory to deal with their datas.And this will cause a
problem that map tasks cost so much memory resource and target data is small.So
I have a idea that wherther we can dynamic set mapTaskAttempt memory size by
its inputDataLength.And this value can be provided by
{{TaskSplitMetaInfo#getInputDataLength}} methods.Besides that,we should
provided a standard unit dataLength for a standard memory size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010781#comment-15010781
]
Hadoop QA commented on MAPREDUCE-6542:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
48s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
40s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 12s
{color} | {color:red} Patch generated 7 new checkstyle issues in root (total
was 299, now 305). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 53s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 26s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 105m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sohaib Iftikhar updated MAPREDUCE-6542:
---
Description:
I use SimpleDateFormat to Parse the JobHistory File before
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I change to use the FastDateFormat to parse the time format and the
time become right
was:
I use SimpleDateFormat to Parse the JobHistory File before
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I chang to use the FastDateFormat to parse the time format and the time
become right
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010517#comment-15010517
]
zhangyubiao commented on MAPREDUCE-6542:
Thanks for [~templedf]'s answer. I update MAPREDUCE-6542-v4.patch for review.
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the
> time become right
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangyubiao updated MAPREDUCE-6542:
---
Attachment: (was: MAPREDUCE-6542-v4.patch)
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the
> time become right
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangyubiao updated MAPREDUCE-6542:
---
Attachment: MAPREDUCE-6542-v4.patch
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the
> time become right
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangyubiao updated MAPREDUCE-6542:
---
Attachment: MAPREDUCE-6542-v4.patch
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the
> time become right
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6551) Dynamic adjust mapTaskAttempt memory size
[
https://issues.apache.org/jira/browse/MAPREDUCE-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated MAPREDUCE-6551:
-
Attachment: MAPREDUCE-6551.001.patch
> Dynamic adjust mapTaskAttempt memory size
> -
>
> Key: MAPREDUCE-6551
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6551
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE-6551.001.patch
>
>
> I found a scenario that the map tasks cost so much resource of cluster.This
> scenario will be happened that if there are many small file blokcs (even some
> are not reach 1M),and this will lead to many map task to read.And in
> gengeral,a map task attempt will use the default config
> {{MRJobConfig#MAP_MEMORY_MB}} to set its resourceCapcity's memory to deal
> with their datas.And this will cause a problem that map tasks cost so much
> memory resource and target data is small.So I have a idea that wherther we
> can dynamic set mapTaskAttempt memory size by its inputDataLength.And this
> value can be provided by {{TaskSplitMetaInfo#getInputDataLength}}
> methods.Besides that,we should provided a standard unit dataLength for a
> standard memory size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6542:
Description:
I use SimpleDateFormat to Parse the JobHistory File before
{code}
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
{code}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I change to use the FastDateFormat to parse the time format and the
time become right
was:
I use SimpleDateFormat to Parse the JobHistory File before
{noformat}
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
{noformat}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I change to use the FastDateFormat to parse the time format and the
time become right
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011090#comment-15011090
]
Daniel Templeton commented on MAPREDUCE-6542:
-
Thanks, [~piaoyu zhang]. Looks good to me. What testing did you do for the
patch?
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6551) Dynamic adjust mapTaskAttempt memory size
[
https://issues.apache.org/jira/browse/MAPREDUCE-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011091#comment-15011091
]
Hadoop QA commented on MAPREDUCE-6551:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 43s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 30s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 18s {color}
| {color:red} hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with
JDK v1.8.0_66 generated 2 new issues (was 360, now 360). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 10s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_85 with JDK v1.7.0_85
generated 3 new issues (was 365, now 365). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 6 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 770, now 775).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 57s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 16s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011094#comment-15011094
]
Daniel Templeton commented on MAPREDUCE-6542:
-
BTW, you should look at the checkstyle comments from the last Jenkins run and
clean those up.
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6551) Dynamic adjust mapTaskAttempt memory size
[
https://issues.apache.org/jira/browse/MAPREDUCE-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010955#comment-15010955
]
Lin Yiqun commented on MAPREDUCE-6551:
--
I add the 3 new config info
* MRJobConfig.MAP_MEMORY_MB_AUTOSET_ENABLED:wherther enable this function
* MRJobConfig.MAP_UNIT_INPUT_LENGTH:the standard unit deal data length.
And if auto-set function is enabled, in
{{MapTaskAttemptImpl#autoSetMemorySize}} method will adjust memory size by its
{{splitInfo}} dataLength.If dataLength is large than UNIT_INPUT_LENGTH,the size
will be larger, other will be smaller.
> Dynamic adjust mapTaskAttempt memory size
> -
>
> Key: MAPREDUCE-6551
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6551
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>
> I found a scenario that the map tasks cost so much resource of cluster.This
> scenario will be happened that if there are many small file blokcs (even some
> are not reach 1M),and this will lead to many map task to read.And in
> gengeral,a map task attempt will use the default config
> {{MRJobConfig#MAP_MEMORY_MB}} to set its resourceCapcity's memory to deal
> with their datas.And this will cause a problem that map tasks cost so much
> memory resource and target data is small.So I have a idea that wherther we
> can dynamic set mapTaskAttempt memory size by its inputDataLength.And this
> value can be provided by {{TaskSplitMetaInfo#getInputDataLength}}
> methods.Besides that,we should provided a standard unit dataLength for a
> standard memory size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6551) Dynamic adjust mapTaskAttempt memory size
[
https://issues.apache.org/jira/browse/MAPREDUCE-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated MAPREDUCE-6551:
-
Status: Patch Available (was: Open)
> Dynamic adjust mapTaskAttempt memory size
> -
>
> Key: MAPREDUCE-6551
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6551
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE-6551.001.patch
>
>
> I found a scenario that the map tasks cost so much resource of cluster.This
> scenario will be happened that if there are many small file blokcs (even some
> are not reach 1M),and this will lead to many map task to read.And in
> gengeral,a map task attempt will use the default config
> {{MRJobConfig#MAP_MEMORY_MB}} to set its resourceCapcity's memory to deal
> with their datas.And this will cause a problem that map tasks cost so much
> memory resource and target data is small.So I have a idea that wherther we
> can dynamic set mapTaskAttempt memory size by its inputDataLength.And this
> value can be provided by {{TaskSplitMetaInfo#getInputDataLength}}
> methods.Besides that,we should provided a standard unit dataLength for a
> standard memory size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6542:
Description:
I use SimpleDateFormat to Parse the JobHistory File before
{noformat}
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
{noformat}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I change to use the FastDateFormat to parse the time format and the
time become right
was:
I use SimpleDateFormat to Parse the JobHistory File before
private static final SimpleDateFormat dateFormat =
new SimpleDateFormat("-MM-dd HH:mm:ss");
public static String getJobDetail(JobInfo job) {
StringBuffer jobDetails = new StringBuffer("");
SummarizedJob ts = new SummarizedJob(job);
jobDetails.append(job.getJobId().toString().trim()).append("\t");
jobDetails.append(job.getUsername()).append("\t");
jobDetails.append(job.getJobname().replaceAll("\\n", "")).append("\t");
jobDetails.append(job.getJobQueueName()).append("\t");
jobDetails.append(job.getPriority()).append("\t");
jobDetails.append(job.getJobConfPath()).append("\t");
jobDetails.append(job.getUberized()).append("\t");
jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
return jobDetails.toString();
}
But I find I query the SubmitTime and LaunchTime in hive and compare JobHistory
File time , I find that the submitTime and launchTime was wrong.
Finally,I change to use the FastDateFormat to parse the time format and the
time become right
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {noformat}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {noformat}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-6550:
Description:
The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
app. When using the DefaultContainerExecutor, this means that the job will
actually run as the Yarn user, so the resulting har files are owned by the Yarn
user instead of the original owner. The permissions are also now world-readable.
In the below example, the archived logs are owned by 'yarn' instead of 'paul'
and are now world-readable:
{noformat}
[root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
...
drwxrwx--- - paul hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005
drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
-rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
-rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
-rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
-rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
drwxrwx--- - paul hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006
-rw-r- 3 paul hadoop 1155 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
-rw-r- 3 paul hadoop 4880 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
...
{noformat}
was:
The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
app. When using the DistributedContainerExecutor, this means that the job will
actually run as the Yarn user, so the resulting har files are owned by the Yarn
user instead of the original owner. The permissions are also now
world-readable.
In the below example, the archived logs are owned by 'yarn' instead of 'paul'
and are now world-readable:
{noformat}
[root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
...
drwxrwx--- - paul hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005
drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
-rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
-rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
-rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
-rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
drwxrwx--- - paul hadoop 0 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006
-rw-r- 3 paul hadoop 1155 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
-rw-r- 3 paul hadoop 4880 2015-10-02 13:24
/tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
...
{noformat}
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6550.001.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn
