Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/9597#issuecomment-156822484
Are you 100% sure that all uses of the partition array only use the index
associated with the individual Partition, and not its position in the array?
At the very least, you should have a test for an RDD with partially empty
partitions and ensure that the indices for hasoffsetranges line up with the
task context partition id.
On Sun, Nov 15, 2015 at 3:48 AM, Saisai Shao <[email protected]>
wrote:
> Hi @koeninger <https://github.com/koeninger> , how about this change?
> Still keeping the mapping relations, so offset range can be retrieved
> through partitionId, just filter out empty partition.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/9597#issuecomment-156794866>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
