Re: I am very new to Spark. I have a very basic question. I have an array of values: listofECtokens: Array[String] = Array(EC-17A5206955089011B, EC-17A5206955089011A) I want to filter an RDD for all o

2015-09-09 Thread Ted Yu 
Prachicsa:
If the number of EC tokens is high, please consider using a set instead of 
array for better lookup performance. 

BTW use short, descriptive subject for future emails. 



> On Sep 9, 2015, at 3:13 AM, Akhil Das  wrote:
> 
> Try this:
> 
> val tocks = Array("EC-17A5206955089011B","EC-17A5206955089011A")
> 
> val rddAll = sc.parallelize(List("This contains EC-17A5206955089011B","This 
> doesnt"))
> 
> rddAll.filter(line => {
> var found = false
> for(item <- tocks){
>if(line.contains(item)) found = true
> }
>found
>   }).collect()
> 
> 
> Output:
> res8: Array[String] = Array(This contains EC-17A5206955089011B)
> 
> Thanks
> Best Regards
> 
>> On Wed, Sep 9, 2015 at 3:25 PM, prachicsa  wrote:
>> 
>> 
>> I am very new to Spark.
>> 
>> I have a very basic question. I have an array of values:
>> 
>> listofECtokens: Array[String] = Array(EC-17A5206955089011B,
>> EC-17A5206955089011A)
>> 
>> I want to filter an RDD for all of these token values. I tried the following
>> way:
>> 
>> val ECtokens = for (token <- listofECtokens) rddAll.filter(line =>
>> line.contains(token))
>> 
>> Output:
>> 
>> ECtokens: Unit = ()
>> 
>> I got an empty Unit even when there are records with these tokens. What am I
>> doing wrong?
>> 
>> 
>> 
>> 
>> --
>> View this message in context: 
>> http://apache-spark-user-list.1001560.n3.nabble.com/I-am-very-new-to-Spark-I-have-a-very-basic-question-I-have-an-array-of-values-listofECtokens-Array-S-tp24617.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>

Re: I am very new to Spark. I have a very basic question. I have an array of values: listofECtokens: Array[String] = Array(EC-17A5206955089011B, EC-17A5206955089011A) I want to filter an RDD for all o

2015-09-09 Thread Akhil Das 
Try this:

val tocks = Array("EC-17A5206955089011B","EC-17A5206955089011A")

val rddAll = sc.parallelize(List("This contains EC-17A5206955089011B","This
doesnt"))

rddAll.filter(line => {
 var found = false
 for(item <- tocks){
if(line.contains(item)) found = true
 }
found
  }).collect()


Output:
res8: Array[String] = Array(This contains EC-17A5206955089011B)

Thanks
Best Regards

On Wed, Sep 9, 2015 at 3:25 PM, prachicsa  wrote:

>
>
> I am very new to Spark.
>
> I have a very basic question. I have an array of values:
>
> listofECtokens: Array[String] = Array(EC-17A5206955089011B,
> EC-17A5206955089011A)
>
> I want to filter an RDD for all of these token values. I tried the
> following
> way:
>
> val ECtokens = for (token <- listofECtokens) rddAll.filter(line =>
> line.contains(token))
>
> Output:
>
> ECtokens: Unit = ()
>
> I got an empty Unit even when there are records with these tokens. What am
> I
> doing wrong?
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/I-am-very-new-to-Spark-I-have-a-very-basic-question-I-have-an-array-of-values-listofECtokens-Array-S-tp24617.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>