Hi, I think that in some (or even all) cases it is possible to push down DISTINCT keyword. However, it is not possible to GROUP BY in general case, unless data is co-located by an attribute mentioned in GROUP BY statement.
On Sat, Mar 24, 2018 at 7:35 PM, besquared <[email protected]> wrote: > Queries involving COUNT(DISTINCT <expr>) could push down the expression + > group by into the mapper. Not doing this This causes an explosion of data > received by the the reducer. On moderately large tables this usually causes > OOM but worse than that it's just infeasible to execute at all. > > In the example below (and in most I can think of at the moment) this could > be rewritten by adding a grouping by the given DISTINCT expression (in this > case account_id). I'd have to think a little more about the more general > relational case but I wanted to post this here to see if there was already > discussion going on about this specific optimization path. > > CREATE TABLE people ( ID LONG PRIMARY KEY, account_id LONG ) > EXPLAIN SELECT COUNT(DISTINCT account_id) AS account_ids FROM people > > Map Phase > > SELECT > __Z0.ACCOUNT_ID AS __C0_0 > FROM PUBLIC.PEOPLE __Z0 > /* PUBLIC.PEOPLE.__SCAN_ */ > > Reduction Phase > > SELECT > COUNT(DISTINCT __C0_0) AS ACCOUNT_IDS > FROM PUBLIC.__T0 > /* PUBLIC."merge_scan" */ > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
