RE: how does isDistinct work on expressions

2016-11-13 Thread assaf.mendelson
Thanks for the pointer. It makes more sense now. Assaf. From: Herman van Hövell tot Westerflier-2 [via Apache Spark Developers List] [mailto:ml-node+s1001551n19842...@n3.nabble.com] Sent: Sunday, November 13, 2016 10:03 PM To: Mendelson, Assaf Subject: Re: how does isDistinct work on expressions

Re: how does isDistinct work on expressions

2016-11-13 Thread Herman van Hövell tot Westerflier
Hi, You should take a look at https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala Spark SQL does not directly support the aggregation of multiple distinct groups. For example select count(distinct a), coun

Re: how does isDistinct work on expressions

2016-11-13 Thread Jacek Laskowski
Hi, I might not have been there yet, but since I'm with the code every day I might be close... When you say "aggregate functions", are you about typed or untyped ones? Just today I reviewed the typed ones and honestly took me some time to figure out what belongs to where. Are you creating a new U