[
https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Hagerf updated SPARK-29427:
-------------------------------------
Description:
The scenario I'm having is that I'm reading two huge bucketed tables and since
a regular join is not performant enough for my cases, I'm using groupByKey to
generate two KeyValueGroupedDatasets and cogroup them to implement the merging
logic I need.
The issue with this approach is that I'm only grouping by the column that the
tables are bucketed by but since I'm using groupByKey the bucketing is
completely ignored and I still get a full shuffle.
What I'm looking for is some functionality to tell Catalyst to group by a
column in a relational way but then give the user a possibility to utilize the
functions of the KeyValueGroupedDataset e.g. cogroup (which is not available
for dataframes)
At current spark (2.4.4) I see no way to do this efficiently. I think this is a
valid use case which if solved would have huge performance benefits.
was:
The scenario I'm having is that I'm reading two huge bucketed tables and since
a regular join is not performant enough for these cases I'm using groupByKey to
generate two KeyValueGroupedDatasets and cogroup them to implement the logic I
need.
The issue with this approach is that I'm only grouping by the column that the
tables are bucketed by but since I'm using groupByKey the bucketing is
completely ignored and I still get a full shuffle.
What I'm looking for is some functionality to tell Catalyst to group by a
column in a relational way but then give the user a possibility to utilize the
functions of the KeyValueGroupedDataset e.g. cogroup (which is not available
for dataframes)
At current spark (2.4.4) I see no way to do this efficiently. I think this is a
valid use case which if solved would have huge performance benefits.
> Create KeyValueGroupedDataset from RelationalGroupedDataset
> -----------------------------------------------------------
>
> Key: SPARK-29427
> URL: https://issues.apache.org/jira/browse/SPARK-29427
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core, SQL
> Affects Versions: 2.4.4
> Reporter: Alexander Hagerf
> Priority: Major
>
> The scenario I'm having is that I'm reading two huge bucketed tables and
> since a regular join is not performant enough for my cases, I'm using
> groupByKey to generate two KeyValueGroupedDatasets and cogroup them to
> implement the merging logic I need.
> The issue with this approach is that I'm only grouping by the column that the
> tables are bucketed by but since I'm using groupByKey the bucketing is
> completely ignored and I still get a full shuffle.
> What I'm looking for is some functionality to tell Catalyst to group by a
> column in a relational way but then give the user a possibility to utilize
> the functions of the KeyValueGroupedDataset e.g. cogroup (which is not
> available for dataframes)
>
> At current spark (2.4.4) I see no way to do this efficiently. I think this is
> a valid use case which if solved would have huge performance benefits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
