[
https://issues.apache.org/jira/browse/KYLIN-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650446#comment-15650446
]
liyang commented on KYLIN-2170:
-------------------------------
Mapper/Reducer is doomed by a classic exception handling problem. E.g. the
below Reducer code, if both reduce() and cleanup() throws exceptions, A will be
discarded and only B is reported, losing a root cause. Some suggest never throw
exception in cleanup() (in the finally block), then job can falsely succeed
when there is no A and only B occurs.
Seems the only solution is log-and-throw in both map()/reduce() and cleanup(),
then in all cases, all exceptions are logged and job will fail on error.
We are refactoring {{KylinMapper}} and {{KylinReducer}} to enforce this design.
{code}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKey()) {
reduce(context.getCurrentKey(), context.getValues(), context); //
<======= Exception A
// If a back up store is used, reset it
Iterator<VALUEIN> iter = context.getValues().iterator();
if(iter instanceof ReduceContext.ValueIterator) {
((ReduceContext.ValueIterator<VALUEIN>)iter).resetBackupStore();
}
}
} finally {
cleanup(context); // <====== Exception B
}
}
{code}
> Mapper/Reducer cleanup() exception handling
> -------------------------------------------
>
> Key: KYLIN-2170
> URL: https://issues.apache.org/jira/browse/KYLIN-2170
> Project: Kylin
> Issue Type: Improvement
> Reporter: liyang
>
> Or it could override the real exception happened in mapper() or reducer()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
