As I understand it, Spark 1.6 changes the behaviour of the first and last aggregate functions to take nulls into account <https://issues.apache.org/jira/browse/SPARK-9740> (where they were ignored in 1.5). From SQL you can use "IGNORE NULLS" to get the old behaviour back. How do I ignore nulls from the Java API <https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/sql/functions.html#first(org.apache.spark.sql.Column)> ? I can't see any way to pass that option, so I suspect I may need to write a user defined aggregate function, but I'd prefer not to if possible.
Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-ignoreNulls-in-first-last-aggregate-functions-tp26031.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
