[
https://issues.apache.org/jira/browse/SPARK-13999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-13999:
------------------------------
Component/s: SQL
> Run 'group by' before building cube
> ------------------------------------
>
> Key: SPARK-13999
> URL: https://issues.apache.org/jira/browse/SPARK-13999
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: lichenglin
>
> When I'm trying to build a cube on a data set witch has about 1 billion count.
> The cube has 7 dimensions.
> It takes a whole day to finish the job with 16 cores;
> Then I run the 'select count (1) from table group by A,B,C,D,E,F,G' first
> and run the cube with the 'group by' result data set.
> The dimensions is the same as 'group by' and do sum on 'count'.
> It just need 45 minutes.
> the group by will reduce the data set's count from billions to millions.
> This depends on the number of dimension.
> We can try in the new version.
> The process of averaging may be complex.Should get the sum and count during
> the group by .
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
