Pyspark RDD search
I have an RDD which is list of list And another RDD which is list of pairs No duplicates in inner list of first RDD and No duplicates in the pairs from second rdd I am trying to check if any pair of second RDD is present in the any list of -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-RDD-search-tp23386.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Pyspark combination
I have an RDD with more than 1000 elements. I have to form the combinations of elements. I tried to use Cartesian transformation and then filtering them. But failing with eof error. Is there any other way to do the same using partitions I am using pyspark -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-combination-tp23385.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark 1.2 compatibility
Yes it works with 2.2 but we are trying to use spark 1.2 on HDP 2.1 On Sat, Jan 17, 2015, 11:18 AM Chitturi Padma [via Apache Spark User List] < ml-node+s1001560n21208...@n3.nabble.com> wrote: > It worked for me. spark 1.2.0 with hadoop 2.2.0 > > On Sat, Jan 17, 2015 at 9:39 PM, bhavyateja [via Apache Spark User List] > <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=21208&i=0>> wrote: > >> Hi all, >> >> Thanks for your contribution. We have checked and confirmed that HDP 2.1 >> YARN don't work with Spark 1.2 >> > On Sat, Jan 17, 2015 at 9:11 AM, bhavya teja potineni <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=21207&i=0>> wrote: >> > Hi >>> >>> Did you try using spark 1.2 on hdp 2.1 YARN >>> >>> Can you please go thru the thread >>> >>> http://apache-spark-user-list.1001560.n3.nabble.com/Troubleshooting-Spark-tt21189.html >>> and check where I am going wrong. As my word count program is erroring >>> out when using spark 1.2 using YARN but its getting executed using spark >>> 0.9.1 >>> >> >>> On Sat, Jan 17, 2015 at 5:55 AM, Chitturi Padma [via Apache Spark User >>> List] <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=21207&i=1>> wrote: >>> >>>> Yes. I built spar 1.2 with apache hadoop 2.2. No compatibility issues. >>>> >>>> On Sat, Jan 17, 2015 at 4:47 AM, bhavyateja [via Apache Spark User >>>> List] <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=21202&i=0>> wrote: >>>> >>>>> Is spark 1.2 is compatibly with HDP 2.1 >>>>> >>>>> -- >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197.html >>>>> To start a new topic under Apache Spark User List, email [hidden >>>>> email] <http:///user/SendEmail.jtp?type=node&node=21202&i=1> >>>>> To unsubscribe from Apache Spark User List, click here. >>>>> NAML >>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>> >>>> >>>> >>>> -- >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> >>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21202.html >>>> To unsubscribe from spark 1.2 compatibility, click here. >>>> NAML >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> If you reply to this email, your message will be added to the >> discussion below: >> > >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21207.html >> > To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=21208&i=1> >> To unsubscribe from Apache Spark User List, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21208.html > To unsubscribe from spark 1.2 compatibility, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=21197&code=YmhhdnlhdGVqYS5wb3RpbmVuaUBnbWFpbC5jb218MjExOTd8LTE1MDEzODgyODY=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21209.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: spark 1.2 compatibility
Hi all, Thanks for your contribution. We have checked and confirmed that HDP 2.1 YARN don't work with Spark 1.2 On Sat, Jan 17, 2015 at 9:11 AM, bhavya teja potineni < bhavyateja.potin...@gmail.com> wrote: > Hi > > Did you try using spark 1.2 on hdp 2.1 YARN > > Can you please go thru the thread > > http://apache-spark-user-list.1001560.n3.nabble.com/Troubleshooting-Spark-tt21189.html > and check where I am going wrong. As my word count program is erroring out > when using spark 1.2 using YARN but its getting executed using spark 0.9.1 > > On Sat, Jan 17, 2015 at 5:55 AM, Chitturi Padma [via Apache Spark User > List] wrote: > >> Yes. I built spar 1.2 with apache hadoop 2.2. No compatibility issues. >> >> On Sat, Jan 17, 2015 at 4:47 AM, bhavyateja [via Apache Spark User List] >> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=21202&i=0>> >> wrote: >> >>> Is spark 1.2 is compatibly with HDP 2.1 >>> >>> -- >>> If you reply to this email, your message will be added to the >>> discussion below: >>> >>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197.html >>> To start a new topic under Apache Spark User List, email [hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=21202&i=1> >>> To unsubscribe from Apache Spark User List, click here. >>> NAML >>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> >> >> -- >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21202.html >> To unsubscribe from spark 1.2 compatibility, click here >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=21197&code=YmhhdnlhdGVqYS5wb3RpbmVuaUBnbWFpbC5jb218MjExOTd8LTE1MDEzODgyODY=> >> . >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21207.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: spark 1.2 compatibility
Hi Did you try using spark 1.2 on hdp 2.1 YARN Can you please go thru the thread http://apache-spark-user-list.1001560.n3.nabble.com/Troubleshooting-Spark-tt21189.html and check where I am going wrong. As my word count program is erroring out when using spark 1.2 using YARN but its getting executed using spark 0.9.1 On Sat, Jan 17, 2015 at 5:55 AM, Chitturi Padma [via Apache Spark User List] wrote: > Yes. I built spar 1.2 with apache hadoop 2.2. No compatibility issues. > > On Sat, Jan 17, 2015 at 4:47 AM, bhavyateja [via Apache Spark User List] > <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=21202&i=0>> wrote: > >> Is spark 1.2 is compatibly with HDP 2.1 >> >> -- >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197.html >> To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=21202&i=1> >> To unsubscribe from Apache Spark User List, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21202.html > To unsubscribe from spark 1.2 compatibility, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=21197&code=YmhhdnlhdGVqYS5wb3RpbmVuaUBnbWFpbC5jb218MjExOTd8LTE1MDEzODgyODY=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197p21205.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
spark 1.2 compatibility
Is spark 1.2 is compatibly with HDP 2.1 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-compatibility-tp21197.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
