Queries involving COUNT(DISTINCT <expr>) could push down the expression +
group by into the mapper. Not doing this This causes an explosion of data
received by the the reducer. On moderately large tables this usually causes
OOM but worse than that it's just infeasible to execute at all.
In the example below (and in most I can think of at the moment) this could
be rewritten by adding a grouping by the given DISTINCT expression (in this
case account_id). I'd have to think a little more about the more general
relational case but I wanted to post this here to see if there was already
discussion going on about this specific optimization path.
CREATE TABLE people ( ID LONG PRIMARY KEY, account_id LONG )
EXPLAIN SELECT COUNT(DISTINCT account_id) AS account_ids FROM people
Map Phase
SELECT
__Z0.ACCOUNT_ID AS __C0_0
FROM PUBLIC.PEOPLE __Z0
/* PUBLIC.PEOPLE.__SCAN_ */
Reduction Phase
SELECT
COUNT(DISTINCT __C0_0) AS ACCOUNT_IDS
FROM PUBLIC.__T0
/* PUBLIC."merge_scan" */
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
