Radovan Zvoncek created CASSANDRA-8367:
------------------------------------------
Summary: Clash between Cassandra and Crunch mapreduce config
Key: CASSANDRA-8367
URL: https://issues.apache.org/jira/browse/CASSANDRA-8367
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Reporter: Radovan Zvoncek
Priority: Minor
We would like to use Cassandra's (Cql)BulkOutputFormats to implement Resource
IOs for Crunch. We want to do this to allow Crunch users write results of their
jobs directly to Cassandra (thus skipping writing them to file system).
In the process of doing this, we found out there is a clash in the mapreduce
job config. The affected config key is 'mapreduce.output.basename'. Cassandra
is using it [1] for something different than Crunch [2]. This is resulting in
some obscure behavior I personally don't understand, but it causes the jobs to
fail.
We went ahead and re-implemented the output format classes to use different
config key, but we'd very much like to stop using them.
[1]
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hadoop/ConfigHelper.java#L54
[2]
https://github.com/apache/crunch/blob/3f13ee65c9debcf6bd7366607f58beae6c73ffe2/crunch-core/src/main/java/org/apache/crunch/io/CrunchOutputs.java#L99
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
