[
https://issues.apache.org/jira/browse/CRUNCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills updated CRUNCH-489:
------------------------------
Attachment: CRUNCH-489d.patch
[~gabriel.reid] thanks for the review-- this patch addresses most of your
comments. A couple of notes:
0) The import o.a.c.* was a bad default on IntelliJ on my new laptop-- all
fixed.
1) NLineFileIT is still testing NLineInputSource by way of the create()
interface, which uses an NLineInputSource for Writable text files.
2) I really like the idea of breaking up the contents of the Iterable into
individual files and disabling combine file to make the parallelization work. I
suppose I could either check to see if the Iterable was a Collection, and if so
cache it into memory and split up the values into chunks, or if it was not, I
could open P connections to HDFS and write one value into each output handle as
I stream through the Iterable. What do you think here?
> Add methods to create PCollections from Java Iterable to Pipeline interface
> ---------------------------------------------------------------------------
>
> Key: CRUNCH-489
> URL: https://issues.apache.org/jira/browse/CRUNCH-489
> Project: Crunch
> Issue Type: Bug
> Reporter: Josh Wills
> Attachments: CRUNCH-489.patch, CRUNCH-489b.patch, CRUNCH-489c.patch,
> CRUNCH-489d.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
