[jira] [Commented] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default

2020-09-23 Thread Jim Brennan (Jira) 


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201046#comment-17201046
 ] 

Jim Brennan commented on MAPREDUCE-7282:


I am -1 on the proposal to remove the v2 algorithm.   Please see [~jlowe]'s 
 
[comment|https://issues.apache.org/jira/browse/MAPREDUCE-4815?focusedCommentId=14271115=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14271115]
 from the original discussion on [MAPREDUCE-4815].

We have been running with the v2 algorithm in production on large clusters for 
years at Verizon Media (Yahoo).   I don't think it is appropriate to remove it.

> MR v2 commit algorithm should be deprecated and not the default
> ---
>
> Key: MAPREDUCE-7282
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1
>Reporter: Steve Loughran
>Priority: Major
>
> The v2 MR commit algorithm moves files from the task attempt dir into the 
> dest dir on task commit -one by one
> It is therefore not atomic
> # if a task commit fails partway through and another task attempt commits 
> -unless exactly the same filenames are used, output of the first attempt may 
> be included in the final result
> # if a worker partitions partway through task commit, and then continues 
> after another attempt has committed, it may partially overwrite the output 
> -even when the filenames are the same
> Both MR and spark assume that task commits are atomic. Either they need to 
> consider that this is not the case, we add a way to probe for a committer 
> supporting atomic task commit, and the engines both add handling for task 
> commit failures (probably fail job)
> Better: we remove this as the default, maybe also warn when it is being used



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7297) Exception thrown in log when trying to download conf from JHS

2020-09-23 Thread Bilwa S T (Jira) 
Bilwa S T created MAPREDUCE-7297:


 Summary: Exception thrown in log when trying to download conf from 
JHS
 Key: MAPREDUCE-7297
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7297
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Bilwa S T
Assignee: Bilwa S T


Below exception is thrown in JHS logs

{code:java}
2020-09-23 21:53:07,437 | ERROR | qtp1635772897-51 | error handling URI: 
/jobhistory/downloadconf/job_1600668504751_0001 | Dispatcher.java:175
java.lang.IllegalStateException: No view rendered for 200
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:171)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at 
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
at 
org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
at 
com.huawei.hadoop.adapter.sso.LogoutFilter.doFilter(LogoutFilter.java:62)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
at 
com.huawei.hadoop.adapter.sso.RefererCheckFilter.doFilter(RefererCheckFilter.java:76)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
at 
org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:70)
at 
com.huawei.hadoop.adapter.sso.HttpServletRequestWrapperFilterWrapper.doFilter(HttpServletRequestWrapperFilterWrapper.java:75)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1591)
at 
org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:238)
at 
com.huawei.hadoop.adapter.sso.Cas20ProxyReceivingTicketValidationFilterWrapper.doFilter(Cas20ProxyReceivingTicketValidationFilterWrapper.java:40)
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org