Hey Jeff, so the reason for copying the task logs is indeed better error-diagnostic. MapReduce/Tez usually only reported 30 to 50% of the exceptions to the client in my experience. So if all of TEZ-1240 is done that might not be much an issue any more…
thanks Johannes On 08 Oct 2014, at 08:24, Jianfeng (Jeff) Zhang <[email protected]> wrote: > Hi Johannes, > > Currently you can see the diagnostics in client if the task fails in > processor, is that what you want ? > > Here's the jira tracking this, https://issues.apache.org/jira/browse/TEZ-1240 > > If you find any exception that is not caught, please create a ticket under > this. > > > > Best Regards, > Jeff Zhang > > > On Wed, Oct 8, 2014 at 2:08 PM, Johannes Zillmann <[email protected]> > wrote: > Hey Mr. Zang, > > so main use case is fetching the tasks logs in case the task fails. I can do > that in a try-catch blog in the processor itself but it has 2 disadvantages: > - log might not be complete > - in case the task fails in not user provided code (outside of the > processor), we don’t capture it > > Johannes > > On 08 Oct 2014, at 01:58, Jianfeng (Jeff) Zhang <[email protected]> > wrote: > > > Hi Johannes, > > > > You can do some post task in Processor, please refer SimpleProcessor which > > has method postOp(), but can do a limited things and can not do things like > > accessing task logs. > > Could you let us know your purpose on customize the post task hook ? > > > > > > Best Regards, > > Jeff Zhang > > > > > > On Thu, Oct 2, 2014 at 4:20 PM, Johannes Zillmann > > <[email protected]> wrote: > > Hey guys, > > > > is there any post task hook in Tez (like the OutputCommitter in MapReduce > > had) ? > > Like to do certain actions (like accessing task logs) once the task > > completes, whether or not the task was successful and whether or not the > > user provided processor got executed or the task failed even before. > > > > Johannes > > > > > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender immediately > > and delete it from your system. Thank You. > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You.
