Re: Create a Column expression from a String

2016-11-21 Thread Stuart White
Yes, that's what I was looking for. Thanks! On Mon, Nov 21, 2016 at 6:56 PM, Michael Armbrust wrote: > You are looking for org.apache.spark.sql.functions.expr() > > On Sat, Nov 19, 2016 at 6:12 PM, Stuart White > wrote: >> >> I'd like to allow for runtime-configured Column expressions in my >>

Re: Create a Column expression from a String

2016-11-21 Thread Michael Armbrust
You are looking for org.apache.spark.sql.functions.expr() On Sat, Nov 19, 2016 at 6:12 PM, Stuart White wrote: > I'd like to allow for runtime-configured Column expressions in my > Spark SQL application. For example, if my application needs a 5-digit > zip code, but the file I'm processing cont

Re: Create a Column expression from a String

2016-11-19 Thread Luciano Resende
Are you looking for UDFs? https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-udfs.html On Sun, Nov 20, 2016 at 3:12 AM Stuart White wrote: > I'd like to allow for runtime-configured Column expressions in my > Spark SQL application. For example, if my application needs

Create a Column expression from a String

2016-11-19 Thread Stuart White
I'd like to allow for runtime-configured Column expressions in my Spark SQL application. For example, if my application needs a 5-digit zip code, but the file I'm processing contains a 9-digit zip code, I'd like to be able to configure my application with the expression "substring('zipCode, 0, 5)"