[
https://issues.apache.org/jira/browse/BIGTOP-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated BIGTOP-1066:
-----------------------------
Description:
The use of old tests (i.e. those that use mapred) can cause issues. Is is it
considered standard pratcice to still support the mapred.* classes?
I say this because in my bigtop smokes, we fail the multifilewc test, because
its RecordReader seems to try to call getPos() after the underlying stream is
closed in the FileSystem implementation. If we update to using the new
mapreduce.* implementation of multifilewc, which uses CombineFileSplit instead
of the older MultiFileInputFormat, the job works fine.
So - this is partly a question : should bigtop tests which rely on mapred.*
1) classes be separated out into a different test suite, or
2) deleted entirely for 2.0 (forward), or
3) simply ignored on a case-by-case/cluster-by-cluster basis?
I favor (1) : It would be nice if we could decide, at runtime, declaratively -
wether or not we wanted to use tests that exersize the old (2009) mapred
classes.
was:
The use of old tests (i.e. those that use mapred) can cause issues. Is is it
considered standard pratcice to still support the mapred.* classes?
I say this because in my bigtop smokes, we fail the multifilewc test, because
its RecordReader seems to try to call getPos() after the underlying stream is
closed in the FileSystem implementation. If we update to using the new
mapreduce.* implementation of multifilewc, which uses CombineFileSplit instead
of the older MultiFileInputFormat, the job works fine.
So - this is partly a question : should bigtop tests which rely on mapred.* 1)
classes be separated out into a different test suite, or
2) deleted entirely for 2.0 (forward), or
3) simply ignored on a case-by-case/cluster-by-cluster basis?
I favor (1) : It would be nice if we could decide, at runtime, declaratively -
wether or not we wanted to use tests that exersize the old (2009) mapred
classes.
> multifilewc and possibly other tests rely on mapred: should they be separated
> or replaced?
> ------------------------------------------------------------------------------------------
>
> Key: BIGTOP-1066
> URL: https://issues.apache.org/jira/browse/BIGTOP-1066
> Project: Bigtop
> Issue Type: Bug
> Reporter: jay vyas
> Priority: Minor
>
> The use of old tests (i.e. those that use mapred) can cause issues. Is is it
> considered standard pratcice to still support the mapred.* classes?
> I say this because in my bigtop smokes, we fail the multifilewc test, because
> its RecordReader seems to try to call getPos() after the underlying stream is
> closed in the FileSystem implementation. If we update to using the new
> mapreduce.* implementation of multifilewc, which uses CombineFileSplit
> instead of the older MultiFileInputFormat, the job works fine.
> So - this is partly a question : should bigtop tests which rely on mapred.*
> 1) classes be separated out into a different test suite, or
> 2) deleted entirely for 2.0 (forward), or
> 3) simply ignored on a case-by-case/cluster-by-cluster basis?
> I favor (1) : It would be nice if we could decide, at runtime, declaratively
> - wether or not we wanted to use tests that exersize the old (2009) mapred
> classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
