Re: Will higher order functions in spark SQL be pushed upstream?

[Michael Davies]

Hi Herman, 

That’s great and thanks for quick reply. The JIRA has an example of transform 
and refers to a presto doc with lots of functions. Do you know which functions 
will be supported. I am interested in using filter for example.

Cheers

Mick


> On 19 Apr 2018, at 10:46, Herman van Hövell tot Westerflier 
>  wrote:
> 
> Hi Mick,
> 
> Higher order functions will be in the next release of Spark (2.4). See 
> SPARK-23908  for more 
> information. The PR should be out in a few weeks.
> 
> - Herman
> 
> On Thu, Apr 19, 2018 at 11:20 AM, Mick Davies  > wrote:
> Hi,
> Regarding higher order functions
> 
> > Yes, we intend to contribute this to open source.
> 
> It doesn't look like this is in 2.3.0, at least I can't find it.
> 
> Do you know when it might reach open source.
> 
> Thanks
> 
> Mick
> 
> 
> 
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ 
> 
> 
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org 
> 
> 
> 

Re: Optimize encoding/decoding strings when using Parquet

[Michael Davies]

Added PR https://github.com/apache/spark/pull/4139 
https://github.com/apache/spark/pull/4139 - I think tests have been 
re-arranged so merge necessary

Mick


 On 19 Jan 2015, at 18:31, Reynold Xin r...@databricks.com wrote:
 
 Definitely go for a pull request!
 
 
 On Mon, Jan 19, 2015 at 10:10 AM, Mick Davies michael.belldav...@gmail.com 
 mailto:michael.belldav...@gmail.com wrote:
 
 Looking at Parquet code - it looks like hooks are already in place to
 support this.
 
 In particular PrimitiveConverter has methods hasDictionarySupport and
 addValueFromDictionary for this purpose. These are not used by
 CatalystPrimitiveConverter.
 
 I think that it would be pretty straightforward to add this. Has anyone
 considered this? Shall I get a pull request  together for it.
 
 Mick
 
 
 
 --
 View this message in context: 
 http://apache-spark-developers-list.1001551.n3.nabble.com/Optimize-encoding-decoding-strings-when-using-Parquet-tp10141p10195.html
  
 http://apache-spark-developers-list.1001551.n3.nabble.com/Optimize-encoding-decoding-strings-when-using-Parquet-tp10141p10195.html
 Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
 mailto:dev-unsubscr...@spark.apache.org
 For additional commands, e-mail: dev-h...@spark.apache.org 
 mailto:dev-h...@spark.apache.org