Re: Spark SQL : how to find element where a field is in a given set
did you create SQLContext?
On Sat, Nov 1, 2014 at 7:51 PM, abhinav chowdary wrote:
> I have same requirement of passing list of values to in clause, when i am
> trying to do
>
> i am getting below error
>
> scala> val longList = Seq[Expression]("a", "b")
> :11: error: type mismatch;
> found : String("a")
> required: org.apache.spark.sql.catalyst.expressions.Expression
>val longList = Seq[Expression]("a", "b")
>
> Thanks
>
>
> On Fri, Aug 29, 2014 at 3:52 PM, Michael Armbrust
> wrote:
>
>> This feature was not part of that version. It will be in 1.1.
>>
>>
>> On Fri, Aug 29, 2014 at 12:33 PM, Jaonary Rabarisoa
>> wrote:
>>
>>>
>>> 1.0.2
>>>
>>>
>>> On Friday, August 29, 2014, Michael Armbrust
>>> wrote:
>>>
What version are you using?
On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa
wrote:
> Still not working for me. I got a compilation error : *value in is
> not a member of Symbol.* Any ideas ?
>
>
> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust <
> mich...@databricks.com> wrote:
>
>> To pass a list to a variadic function you can use the type ascription
>> :_*
>>
>> For example:
>>
>> val longList = Seq[Expression]("a", "b", ...)
>> table("src").where('key in (longList: _*))
>>
>> Also, note that I had to explicitly specify Expression as the type
>> parameter of Seq to ensure that the compiler converts "a" and "b" into
>> Spark SQL expressions.
>>
>>
>>
>>
>> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa <
>> jaon...@gmail.com> wrote:
>>
>>> ok, but what if I have a long list do I need to hard code like this
>>> every element of my list of is there a function that translate a list
>>> into
>>> a tuple ?
>>>
>>>
>>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
>>> mich...@databricks.com> wrote:
>>>
You don't need the Seq, as in is a variadic function.
personTable.where('name in ("foo", "bar"))
On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa <
jaon...@gmail.com> wrote:
> Hi all,
>
> What is the expression that I should use with spark sql DSL if I
> need to retreive
> data with a field in a given set.
> For example :
>
> I have the following schema
>
> case class Person(name: String, age: Int)
>
> And I need to do something like :
>
> personTable.where('name in Seq("foo", "bar")) ?
>
>
> Cheers.
>
>
> Jaonary
>
>>>
>>
>
>>>
>>
>>
>
>
> --
> Warm Regards
> Abhinav Chowdary
>
Re: Spark SQL : how to find element where a field is in a given set
I have same requirement of passing list of values to in clause, when i am
trying to do
i am getting below error
scala> val longList = Seq[Expression]("a", "b")
:11: error: type mismatch;
found : String("a")
required: org.apache.spark.sql.catalyst.expressions.Expression
val longList = Seq[Expression]("a", "b")
Thanks
On Fri, Aug 29, 2014 at 3:52 PM, Michael Armbrust
wrote:
> This feature was not part of that version. It will be in 1.1.
>
>
> On Fri, Aug 29, 2014 at 12:33 PM, Jaonary Rabarisoa
> wrote:
>
>>
>> 1.0.2
>>
>>
>> On Friday, August 29, 2014, Michael Armbrust
>> wrote:
>>
>>> What version are you using?
>>>
>>>
>>>
>>> On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa
>>> wrote:
>>>
Still not working for me. I got a compilation error : *value in is not
a member of Symbol.* Any ideas ?
On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust <
mich...@databricks.com> wrote:
> To pass a list to a variadic function you can use the type ascription
> :_*
>
> For example:
>
> val longList = Seq[Expression]("a", "b", ...)
> table("src").where('key in (longList: _*))
>
> Also, note that I had to explicitly specify Expression as the type
> parameter of Seq to ensure that the compiler converts "a" and "b" into
> Spark SQL expressions.
>
>
>
>
> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa > wrote:
>
>> ok, but what if I have a long list do I need to hard code like this
>> every element of my list of is there a function that translate a list
>> into
>> a tuple ?
>>
>>
>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
>> mich...@databricks.com> wrote:
>>
>>> You don't need the Seq, as in is a variadic function.
>>>
>>> personTable.where('name in ("foo", "bar"))
>>>
>>>
>>>
>>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa <
>>> jaon...@gmail.com> wrote:
>>>
Hi all,
What is the expression that I should use with spark sql DSL if I
need to retreive
data with a field in a given set.
For example :
I have the following schema
case class Person(name: String, age: Int)
And I need to do something like :
personTable.where('name in Seq("foo", "bar")) ?
Cheers.
Jaonary
>>>
>>>
>>
>
>>>
>>
>
>
--
Warm Regards
Abhinav Chowdary
Re: Spark SQL : how to find element where a field is in a given set
This feature was not part of that version. It will be in 1.1.
On Fri, Aug 29, 2014 at 12:33 PM, Jaonary Rabarisoa
wrote:
>
> 1.0.2
>
>
> On Friday, August 29, 2014, Michael Armbrust
> wrote:
>
>> What version are you using?
>>
>>
>>
>> On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa
>> wrote:
>>
>>> Still not working for me. I got a compilation error : *value in is not
>>> a member of Symbol.* Any ideas ?
>>>
>>>
>>> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust <
>>> mich...@databricks.com> wrote:
>>>
To pass a list to a variadic function you can use the type ascription
:_*
For example:
val longList = Seq[Expression]("a", "b", ...)
table("src").where('key in (longList: _*))
Also, note that I had to explicitly specify Expression as the type
parameter of Seq to ensure that the compiler converts "a" and "b" into
Spark SQL expressions.
On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa
wrote:
> ok, but what if I have a long list do I need to hard code like this
> every element of my list of is there a function that translate a list into
> a tuple ?
>
>
> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
> mich...@databricks.com> wrote:
>
>> You don't need the Seq, as in is a variadic function.
>>
>> personTable.where('name in ("foo", "bar"))
>>
>>
>>
>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa > > wrote:
>>
>>> Hi all,
>>>
>>> What is the expression that I should use with spark sql DSL if I
>>> need to retreive
>>> data with a field in a given set.
>>> For example :
>>>
>>> I have the following schema
>>>
>>> case class Person(name: String, age: Int)
>>>
>>> And I need to do something like :
>>>
>>> personTable.where('name in Seq("foo", "bar")) ?
>>>
>>>
>>> Cheers.
>>>
>>>
>>> Jaonary
>>>
>>
>>
>
>>>
>>
>
Re: Spark SQL : how to find element where a field is in a given set
1.0.2
On Friday, August 29, 2014, Michael Armbrust wrote:
> What version are you using?
>
>
>
> On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa > wrote:
>
>> Still not working for me. I got a compilation error : *value in is not a
>> member of Symbol.* Any ideas ?
>>
>>
>> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust > > wrote:
>>
>>> To pass a list to a variadic function you can use the type ascription
>>> :_*
>>>
>>> For example:
>>>
>>> val longList = Seq[Expression]("a", "b", ...)
>>> table("src").where('key in (longList: _*))
>>>
>>> Also, note that I had to explicitly specify Expression as the type
>>> parameter of Seq to ensure that the compiler converts "a" and "b" into
>>> Spark SQL expressions.
>>>
>>>
>>>
>>>
>>> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa >> > wrote:
>>>
ok, but what if I have a long list do I need to hard code like this
every element of my list of is there a function that translate a list into
a tuple ?
On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
mich...@databricks.com
> wrote:
> You don't need the Seq, as in is a variadic function.
>
> personTable.where('name in ("foo", "bar"))
>
>
>
> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa > wrote:
>
>> Hi all,
>>
>> What is the expression that I should use with spark sql DSL if I need
>> to retreive
>> data with a field in a given set.
>> For example :
>>
>> I have the following schema
>>
>> case class Person(name: String, age: Int)
>>
>> And I need to do something like :
>>
>> personTable.where('name in Seq("foo", "bar")) ?
>>
>>
>> Cheers.
>>
>>
>> Jaonary
>>
>
>
>>>
>>
>
Re: Spark SQL : how to find element where a field is in a given set
What version are you using?
On Fri, Aug 29, 2014 at 2:22 AM, Jaonary Rabarisoa
wrote:
> Still not working for me. I got a compilation error : *value in is not a
> member of Symbol.* Any ideas ?
>
>
> On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust
> wrote:
>
>> To pass a list to a variadic function you can use the type ascription :_*
>>
>> For example:
>>
>> val longList = Seq[Expression]("a", "b", ...)
>> table("src").where('key in (longList: _*))
>>
>> Also, note that I had to explicitly specify Expression as the type
>> parameter of Seq to ensure that the compiler converts "a" and "b" into
>> Spark SQL expressions.
>>
>>
>>
>>
>> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa
>> wrote:
>>
>>> ok, but what if I have a long list do I need to hard code like this
>>> every element of my list of is there a function that translate a list into
>>> a tuple ?
>>>
>>>
>>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust <
>>> mich...@databricks.com> wrote:
>>>
You don't need the Seq, as in is a variadic function.
personTable.where('name in ("foo", "bar"))
On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa
wrote:
> Hi all,
>
> What is the expression that I should use with spark sql DSL if I need
> to retreive
> data with a field in a given set.
> For example :
>
> I have the following schema
>
> case class Person(name: String, age: Int)
>
> And I need to do something like :
>
> personTable.where('name in Seq("foo", "bar")) ?
>
>
> Cheers.
>
>
> Jaonary
>
>>>
>>
>
Re: Spark SQL : how to find element where a field is in a given set
Still not working for me. I got a compilation error : *value in is not a
member of Symbol.* Any ideas ?
On Fri, Aug 29, 2014 at 9:46 AM, Michael Armbrust
wrote:
> To pass a list to a variadic function you can use the type ascription :_*
>
> For example:
>
> val longList = Seq[Expression]("a", "b", ...)
> table("src").where('key in (longList: _*))
>
> Also, note that I had to explicitly specify Expression as the type
> parameter of Seq to ensure that the compiler converts "a" and "b" into
> Spark SQL expressions.
>
>
>
>
> On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa
> wrote:
>
>> ok, but what if I have a long list do I need to hard code like this every
>> element of my list of is there a function that translate a list into a
>> tuple ?
>>
>>
>> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust > > wrote:
>>
>>> You don't need the Seq, as in is a variadic function.
>>>
>>> personTable.where('name in ("foo", "bar"))
>>>
>>>
>>>
>>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa
>>> wrote:
>>>
Hi all,
What is the expression that I should use with spark sql DSL if I need
to retreive
data with a field in a given set.
For example :
I have the following schema
case class Person(name: String, age: Int)
And I need to do something like :
personTable.where('name in Seq("foo", "bar")) ?
Cheers.
Jaonary
>>>
>>>
>>
>
Re: Spark SQL : how to find element where a field is in a given set
To pass a list to a variadic function you can use the type ascription :_*
For example:
val longList = Seq[Expression]("a", "b", ...)
table("src").where('key in (longList: _*))
Also, note that I had to explicitly specify Expression as the type
parameter of Seq to ensure that the compiler converts "a" and "b" into
Spark SQL expressions.
On Thu, Aug 28, 2014 at 11:52 PM, Jaonary Rabarisoa
wrote:
> ok, but what if I have a long list do I need to hard code like this every
> element of my list of is there a function that translate a list into a
> tuple ?
>
>
> On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust
> wrote:
>
>> You don't need the Seq, as in is a variadic function.
>>
>> personTable.where('name in ("foo", "bar"))
>>
>>
>>
>> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa
>> wrote:
>>
>>> Hi all,
>>>
>>> What is the expression that I should use with spark sql DSL if I need to
>>> retreive
>>> data with a field in a given set.
>>> For example :
>>>
>>> I have the following schema
>>>
>>> case class Person(name: String, age: Int)
>>>
>>> And I need to do something like :
>>>
>>> personTable.where('name in Seq("foo", "bar")) ?
>>>
>>>
>>> Cheers.
>>>
>>>
>>> Jaonary
>>>
>>
>>
>
Re: Spark SQL : how to find element where a field is in a given set
ok, but what if I have a long list do I need to hard code like this every
element of my list of is there a function that translate a list into a
tuple ?
On Fri, Aug 29, 2014 at 3:24 AM, Michael Armbrust
wrote:
> You don't need the Seq, as in is a variadic function.
>
> personTable.where('name in ("foo", "bar"))
>
>
>
> On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa
> wrote:
>
>> Hi all,
>>
>> What is the expression that I should use with spark sql DSL if I need to
>> retreive
>> data with a field in a given set.
>> For example :
>>
>> I have the following schema
>>
>> case class Person(name: String, age: Int)
>>
>> And I need to do something like :
>>
>> personTable.where('name in Seq("foo", "bar")) ?
>>
>>
>> Cheers.
>>
>>
>> Jaonary
>>
>
>
Re: Spark SQL : how to find element where a field is in a given set
You don't need the Seq, as in is a variadic function.
personTable.where('name in ("foo", "bar"))
On Thu, Aug 28, 2014 at 3:09 AM, Jaonary Rabarisoa
wrote:
> Hi all,
>
> What is the expression that I should use with spark sql DSL if I need to
> retreive
> data with a field in a given set.
> For example :
>
> I have the following schema
>
> case class Person(name: String, age: Int)
>
> And I need to do something like :
>
> personTable.where('name in Seq("foo", "bar")) ?
>
>
> Cheers.
>
>
> Jaonary
>
Spark SQL : how to find element where a field is in a given set
Hi all,
What is the expression that I should use with spark sql DSL if I need to
retreive
data with a field in a given set.
For example :
I have the following schema
case class Person(name: String, age: Int)
And I need to do something like :
personTable.where('name in Seq("foo", "bar")) ?
Cheers.
Jaonary
