[
https://issues.apache.org/jira/browse/SPARK-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394231#comment-14394231
]
Sean Owen commented on SPARK-6664:
----------------------------------
Yes _k_ estimates is better than 1; this is both more expensive and less
important when the data size is large. But, yes it has value and I can see the
argument that it's more important if the response is actually time-dependent. I
wasn't suggesting that {{MLUtils.kFold}} implements this, but that it was a
related piece of code. If ordering matters and the input has an ordering that
biases the result, then yes you would randomly permute the partition or RDD.
This isn't true for every algorithm but for some.
Same thing here really, I think you can order, bucket by range, and union in
the straightforward way and it will be as performant as anything I can think
of. You have to write some code, but it's flexible.
The question is how much it's worth adding another method versus how often this
is used. I can see this being useful for time series. I suppose that if it
turns out there's a much fast-er way to do this but it's complex, and it is
used, then it does need to be wrapped up in a utility method.
> Split Ordered RDD into multiple RDDs by keys (boundaries or intervals)
> ----------------------------------------------------------------------
>
> Key: SPARK-6664
> URL: https://issues.apache.org/jira/browse/SPARK-6664
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Florian Verhein
>
> I can't find this functionality (if I missed something, apologies!), but it
> would be very useful for evaluating ml models.
> *Use case example*
> suppose you have pre-processed web logs for a few months, and now want to
> split it into a training set (where you train a model to predict some aspect
> of site accesses, perhaps per user) and an out of time test set (where you
> evaluate how well your model performs in the future). This example has just a
> single split, but in general you could want more for cross validation. You
> may also want to have multiple overlaping intervals.
> *Specification*
> 1. Given an Ordered RDD and an ordered sequence of n boundaries (i.e. keys),
> return n+1 RDDs such that values in the ith RDD are within the (i-1)th and
> ith boundary.
> 2. More complex alternative (but similar under the hood): provide a sequence
> of possibly overlapping intervals (ordered by the start key of the interval),
> and return the RDDs containing values within those intervals.
> *Implementation ideas / notes for 1*
> - The ordered RDDs are likely RangePartitioned (or there should be a simple
> way to find ranges from partitions in an ordered RDD)
> - Find the partitions containing the boundary, and split them in two.
> - Construct the new RDDs from the original partitions (and any split ones)
> I suspect this could be done by launching only a few jobs to split the
> partitions containing the boundaries.
> Alternatively, it might be possible to decorate these partitions and use them
> in more than one RDD. I.e. let one of these partitions (for boundary i) be p.
> Apply two decorators p' and p'', where p' is masks out values above the ith
> boundary, and p'' masks out values below the ith boundary. Any operations on
> these partitions apply only to values not masked out. Then assign p' to the
> ith output RDD and p'' to the (i+1)th output RDD.
> If I understand Spark correctly, this should not require any jobs. Not sure
> whether it's worth trying this optimisation.
> *Implementation ideas / notes for 2*
> This is very similar, except that we have to handle entire (or parts) of
> partitions belonging to more than one output RDD, since they are no longer
> mutually exclusive. But since RDDs are immutable(??), the decorator idea
> should still work?
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
