[
https://issues.apache.org/jira/browse/SPARK-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-7150:
-------------------------------
Description:
It would be handy to have easy ways to construct random columns for DataFrames.
Proposed API:
{code}
class SQLContext {
// Return a DataFrame with a single column named "id" that has consecutive
value from 0 to n.
def range(n: Long): DataFrame
}
{code}
Usage:
{code}
// uniform distribution
ctx.range(1000).select(rand())
// normal distribution
ctx.range(1000).select(randn())
{code}
was:
It would be handy to have easy ways to construct random columns for DataFrames.
Proposed API:
{code}
class SQLContext {
// Return a DataFrame with a single column named "id" that has consecutive
value from 0 to n.
def range(n: Long): DataFrame
}
{code}
Usage:
{code}
// uniform distribution
ctx.range(1000).select(rand())
// normal distribution
ctx.range(1000).select(randn())
{code}
This could be part of spark.ml (in which case it could be in a RandomRDD object
resembling the one in spark.mllib), or it could be in SQL proper. I'd go for
spark.ml, but either is fine with me.
> Facilitate random column generation for DataFrames
> --------------------------------------------------
>
> Key: SPARK-7150
> URL: https://issues.apache.org/jira/browse/SPARK-7150
> Project: Spark
> Issue Type: Sub-task
> Components: ML, SQL
> Reporter: Joseph K. Bradley
> Priority: Minor
>
> It would be handy to have easy ways to construct random columns for
> DataFrames. Proposed API:
> {code}
> class SQLContext {
> // Return a DataFrame with a single column named "id" that has consecutive
> value from 0 to n.
> def range(n: Long): DataFrame
> }
> {code}
> Usage:
> {code}
> // uniform distribution
> ctx.range(1000).select(rand())
> // normal distribution
> ctx.range(1000).select(randn())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
