[
https://issues.apache.org/jira/browse/CRUNCH-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695746#comment-14695746
]
Josh Wills commented on CRUNCH-556:
-----------------------------------
Yeah, so I think to [~smungre]'s point, we need to add integration tests
between HBase and Spark if we expect that Crunch-on-Spark will play nice
w/HBase. There's also no way to really test the total sort patch w/o starting
up a mini cluster (I think, could be wrong.) And so I'm procrastinating doing
that by pretending to think really hard about the right way to do it-- new
module? crunch-spark as a test dependency for crunch-hbase? Or crunch-hbase as
a test dependency for crunch-spark?
> Fix total sorts in Crunch-on-Spark
> ----------------------------------
>
> Key: CRUNCH-556
> URL: https://issues.apache.org/jira/browse/CRUNCH-556
> Project: Crunch
> Issue Type: Bug
> Components: Spark
> Affects Versions: 0.13.0
> Reporter: Josh Wills
> Fix For: 0.14.0
>
> Attachments: CRUNCH-556.patch
>
>
> From the user mailing list, trying to perform a total sort to create an HFile
> w/Crunch on Spark throws the following exception:
> https://gist.github.com/anonymous/920c000f20229eaa76d8
> The problem can be traced to not properly configuring the partitioner w/the
> path to the partition file that is stored in the GroupingOptions extra
> configuration settings. These settings get passed correctly for the MR job,
> but not for the Spark ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
