[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-003.patch
Status: Patch Available (was: Open)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch, MAPREDUCE-7304-002.patch,
> MAPREDUCE-7304-003.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Status: Open (was: Patch Available)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch, MAPREDUCE-7304-002.patch,
> MAPREDUCE-7304-003.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7305) [JDK 11] TestMRJobsWithProfiler fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7305:
-
Fix Version/s: 3.3.1
3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.3.
> [JDK 11] TestMRJobsWithProfiler fails
> -
>
> Key: MAPREDUCE-7305
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7305
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Environment: OpenJDK 11.0.9
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/56/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobsWithProfiler/testDefaultProfiler/
> {noformat}
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
> 85.232 s <<< FAILURE! - in
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> [ERROR]
> testDefaultProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)
> Time elapsed: 30.901 s <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDefaultProfiler(TestMRJobsWithProfiler.java:111)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.lang.Thread.run(Thread.java:834)
> [ERROR]
> testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)
> Time elapsed: 27.956 s <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher
[
https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reassigned MAPREDUCE-7307:
--
Assignee: Zhihua Deng
> Potential thread leak in LocatedFileStatusFetcher
> -
>
> Key: MAPREDUCE-7307
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7307
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>
> We see that when using LocatedFileStatusFetcher to get file infos In
> parallel, if the listStatus thread is interrupted, the executor service in
> LocatedFileStatusFetcher is left unclosed, the thread stack will like this:
> {noformat}
> "GetFileInfo #63" #125 daemon prio=5 os_prio=0 tid=0x7f6198106800
> nid=0x881 waiting on condition [0x7f60d9fde000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x82e810a8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This caused by if condition.await() throws InterruptedException, the method
> `shutDownNow` for the executor service would not be called as a result,
> should move such resource releasing call into the finally block.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher
[
https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated MAPREDUCE-7307:
---
Component/s: job submission
> Potential thread leak in LocatedFileStatusFetcher
> -
>
> Key: MAPREDUCE-7307
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7307
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Zhihua Deng
>Priority: Major
>
> We see that when using LocatedFileStatusFetcher to get file infos In
> parallel, if the listStatus thread is interrupted, the executor service in
> LocatedFileStatusFetcher is left unclosed, the thread stack will like this:
> {noformat}
> "GetFileInfo #63" #125 daemon prio=5 os_prio=0 tid=0x7f6198106800
> nid=0x881 waiting on condition [0x7f60d9fde000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x82e810a8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This caused by if condition.await() throws InterruptedException, the method
> `shutDownNow` for the executor service would not be called as a result,
> should move such resource releasing call into the finally block.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher
Zhihua Deng created MAPREDUCE-7307:
--
Summary: Potential thread leak in LocatedFileStatusFetcher
Key: MAPREDUCE-7307
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7307
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhihua Deng
We see that when using LocatedFileStatusFetcher to get file infos In parallel,
if the listStatus thread is interrupted, the executor service in
LocatedFileStatusFetcher is left unclosed, the thread stack will like this:
{noformat}
"GetFileInfo #63" #125 daemon prio=5 os_prio=0 tid=0x7f6198106800 nid=0x881
waiting on condition [0x7f60d9fde000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x82e810a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
This caused by if condition.await() throws InterruptedException, the method
`shutDownNow` for the executor service would not be called as a result, should
move such resource releasing call into the finally block.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6447) reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
[ https://issues.apache.org/jira/browse/MAPREDUCE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233820#comment-17233820 ] Nicholas Gerner commented on MAPREDUCE-6447: In a streaming context I've turned down the Java heap size to about 20% of the container setting, mapreduce.reduce.memory.mb via mapreduce.reduce.java.opts. So how does mapreduce.reduce.shuffle.input.buffer.percent interact with that? Is this setting being used in the same JVM as the reducer runs? It seems like that buffer percent setting is applied to the container's setting, not the Java heap. If that's the case it seems like mapreduce.reduce.shuffle.input.buffer.percent should reflect something like whatever fraction of the container memory size is actually available to the Java heap, right? > reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space" > --- > > Key: MAPREDUCE-6447 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6447 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0, 2.6.0, 2.5.1, 2.7.1 >Reporter: shuzhangyao >Assignee: shuzhangyao >Priority: Minor > > 2015-08-11 14:03:54,550 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in fetcher#10 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56) > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.(InMemoryMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:303) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:293) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233776#comment-17233776
]
Hadoop QA commented on MAPREDUCE-7304:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 15m
19s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
4s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 50s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
11s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/30/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch
generated 6 new + 675 unchanged - 0 fixed = 681 total (was 675) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 34s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Status: Open (was: Patch Available)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch, MAPREDUCE-7304-002.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-002.patch
Status: Patch Available (was: Open)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch, MAPREDUCE-7304-002.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233525#comment-17233525
]
Hadoop QA commented on MAPREDUCE-7304:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
4s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
48s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
43s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 13s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
1s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 25s{color}
|
[/results-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/29/artifact/out/results-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt]
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 generated 1 new + 351
unchanged - 1 fixed = 352 total (was 352) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/29/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch
generated 30 new + 675 unchanged - 0 fixed = 705 total
[jira] [Commented] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233521#comment-17233521
]
Peter Bacsko commented on MAPREDUCE-7304:
-
{{Class.newInstance}} is deprecated, replace that to
{{Class.getDeclaredConstructor().newInstance()}}.
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233517#comment-17233517
]
Peter Bacsko commented on MAPREDUCE-7304:
-
[~zerdmann] please fix the checkstyle issues, there are quite a few.
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233513#comment-17233513
]
Hadoop QA commented on MAPREDUCE-7304:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
0s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
49s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 5s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
3s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 24s{color}
|
[/results-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/28/artifact/out/results-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt]
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 generated 1 new + 352
unchanged - 1 fixed = 353 total (was 353) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/28/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch
generated 30 new + 675 unchanged - 0 fixed = 705 total
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: (was: MAPREDUCE-7304-001.patch)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Status: Open (was: Patch Available)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-001.patch
Status: Patch Available (was: Open)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-001.patch
Status: Patch Available (was: Open)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: (was: MAPREDUCE-7304-001.patch)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: (was: MAPREDUCE-7304-001.diff)
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-001.patch
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.patch
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann updated MAPREDUCE-7304:
--
Attachment: MAPREDUCE-7304-001.diff
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
> Attachments: MAPREDUCE-7304-001.diff
>
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7304) Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class
[
https://issues.apache.org/jira/browse/MAPREDUCE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Erdmann reassigned MAPREDUCE-7304:
-
Assignee: Zoltán Erdmann
> Enhance the map-reduce Job end notifier to be able to notify the given URL
> via a custom class
> -
>
> Key: MAPREDUCE-7304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Daniel Fritsi
>Assignee: Zoltán Erdmann
>Priority: Major
>
> Currently
> {color:#0747a6}{{*org.apache.hadoop.mapreduce.v2.app.JobEndNotifier*}}{color}
> allows a very limited configuration on how the given Job end notification URL
> should be notified. We should enhance this, but instead of adding more
> *{color:#0747A6}{{mapreduce.job.end-notification.*}}{color}* properties to be
> able to configure the underlying HttpURLConnection, we should add a new
> property so users can use their own notifier class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7306) Mistaken cyclic check in JobControl Job DAG
[
https://issues.apache.org/jira/browse/MAPREDUCE-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaomin Zhang updated MAPREDUCE-7306:
-
Description:
MAPREDUCE-4371 added cyclic dependency check for a topologic sorted Job DAG.
However it is somehow not properly implemented for the `sort` phase.
In the while loop as below:
{code:java}
while (!SourceSet.isEmpty()) {
ControlledJob controlledJob = SourceSet.iterator().next();
SourceSet.remove(controlledJob);
if (controlledJob.getDependentJobs() != null) {
for (int i = 0; i < controlledJob.getDependentJobs().size(); i++) {
ControlledJob depenControlledJob =
controlledJob.getDependentJobs().get(i);
processedMap.get(controlledJob).add(depenControlledJob);
if (!hasInComingEdge(controlledJob, jobList, processedMap)) {
SourceSet.add(depenControlledJob);
}
}
}
}{code}
It adds the parent/dependent Job node to the processedMap for current Job node.
And then it's supposed to add the parent/dependent Job node into SourceSet if
the parent Job node is not processed yet or does not have any child node except
the current processed one. However it mistakenly checks the current one:
*hasInComingEdge(controlledJob)* , while adding the parent node to the
SourceSet: *SourceSet.add(depenControlledJob)*
This breaks Job DAGs like below:
job1.addDependingJob(job2);
job2.addDependingJob(job3);
job4.addDependingJob(job2);
Above code reports a cyclic dependency for job2, because job2 is added into
SourceSet twice.
was:
MAPREDUCE-4371 added cyclic dependency check for a topologic sorted Job DAG.
However it is somehow not properly implemented for the `sort` phase.
In the while loop as below:
{code:java}
while (!SourceSet.isEmpty()) {
ControlledJob controlledJob = SourceSet.iterator().next();
SourceSet.remove(controlledJob);
if (controlledJob.getDependentJobs() != null) {
for (int i = 0; i < controlledJob.getDependentJobs().size(); i++) {
ControlledJob depenControlledJob =
controlledJob.getDependentJobs().get(i);
processedMap.get(controlledJob).add(depenControlledJob);
if (!hasInComingEdge(controlledJob, jobList, processedMap)) {
SourceSet.add(depenControlledJob);
}
}
}
}{code}
visit the parent node followed by the child node. If the given graph contains a
cycle, then there is at least one node which is a parent as well as a child so
this will break Topological Order. Therefore, after the topological sort, check
for every directed edge whether it follows the order or not.
> Mistaken cyclic check in JobControl Job DAG
> ---
>
> Key: MAPREDUCE-7306
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7306
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.0.0
>Reporter: Xiaomin Zhang
>Priority: Major
>
> MAPREDUCE-4371 added cyclic dependency check for a topologic sorted Job DAG.
> However it is somehow not properly implemented for the `sort` phase.
> In the while loop as below:
> {code:java}
> while (!SourceSet.isEmpty()) {
>ControlledJob controlledJob = SourceSet.iterator().next();
>SourceSet.remove(controlledJob);
>if (controlledJob.getDependentJobs() != null) {
> for (int i = 0; i < controlledJob.getDependentJobs().size(); i++) {
>ControlledJob depenControlledJob =
> controlledJob.getDependentJobs().get(i);
>processedMap.get(controlledJob).add(depenControlledJob);
>if (!hasInComingEdge(controlledJob, jobList, processedMap)) {
> SourceSet.add(depenControlledJob);
>}
> }
>}
> }{code}
> It adds the parent/dependent Job node to the processedMap for current Job
> node. And then it's supposed to add the parent/dependent Job node into
> SourceSet if the parent Job node is not processed yet or does not have any
> child node except the current processed one. However it mistakenly checks the
> current one: *hasInComingEdge(controlledJob)* , while adding the parent node
> to the SourceSet: *SourceSet.add(depenControlledJob)*
> This breaks Job DAGs like below:
> job1.addDependingJob(job2);
> job2.addDependingJob(job3);
> job4.addDependingJob(job2);
> Above code reports a cyclic dependency for job2, because job2 is added into
> SourceSet twice.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
