[
https://issues.apache.org/jira/browse/SPARK-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576439#comment-14576439
]
Amey Chaugule commented on SPARK-7998:
--------------------------------------
Are you proposing a method freqItemsList that does something like:
df.stat.freqItems(Seq(cols...)).head().toSeq.toList ?
> A better frequent item API
> --------------------------
>
> Key: SPARK-7998
> URL: https://issues.apache.org/jira/browse/SPARK-7998
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Reynold Xin
>
> The current freqItems API is really awkward to use. It returns a DataFrame
> with a single row, in which each value is an array of frequent items.
> This design doesn't work well for exploratory data analysis (running show --
> when there are more than 2 or 3 frequent values, the values get cut off):
> {code}
> In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show()
> +------------------+------------------+-----------------+
> | a_freqItems| b_freqItems| c_freqItems|
> +------------------+------------------+-----------------+
> |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)|
> +------------------+------------------+-----------------+
> {code}
> It also doesn't work well for serious engineering, since it is hard to get
> the value out.
> We should create a new function (so we maintain source/binary compatibility)
> that returns a list of list of values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
