Re: SparkR parallelize not found with 1.4.1?
I forgot to mention that if you need to access these functions for some reason, you can prepend the function call with the SparkR private namespace, like so, SparkR:::lapply(rdd, func). On 6/25/15, 9:30 AM, "Felix C" wrote: >Thanks! It's good to know > >--- Original Message --- > >From: "Eskilson,Aleksander" >Sent: June 25, 2015 5:57 AM >To: "Felix C" , user@spark.apache.org >Subject: Re: SparkR parallelize not found with 1.4.1? > >Hi there, > >Parallelize is part of the RDD API which was made private for Spark v. >1.4.0. Some functions in the RDD API were considered too low-level to >expose, so only most of the DataFrame API is currently public. The >original rationale for this decision can be found on the issue's JIRA [1]. >The devs are still considering which parts of the RDD API, if any, should >be made public for later releases. If you have some use case that you feel >is most easily addressed by the functions currently private in the RDD >API, go ahead and let the dev mailing list know. > >Alek >[1] -- https://issues.apache.org/jira/browse/SPARK-7230 > >On 6/25/15, 12:24 AM, "Felix C" wrote: > >>Hi, >> >>It must be something very straightforward... >> >>Not working: >>parallelize(sc) >>Error: could not find function "parallelize" >> >>Working: >>df <- createDataFrame(sqlContext, localDF) >> >>What did I miss? >>Thanks >>?B�CB >>� >>?�?[��X��ܚX�K??K[XZ[?�?\�\�][��X��ܚX�P?�?\�˘\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[ >>� >>?�??K[XZ[?�?\�\�Z?[???�?\�˘\?X�?K�ܙ�B�B > > >CONFIDENTIALITY NOTICE This message and any included attachments are from >Cerner Corporation and are intended only for the addressee. The >information contained in this message is confidential and may constitute >inside or non-public information under international, federal, or state >securities laws. Unauthorized forwarding, printing, copying, >distribution, or use of such information is strictly prohibited and may >be unlawful. If you are not the addressee, please promptly delete this >message and notify the sender of the delivery error by e-mail or you may >call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) >(816)221-1024. >?B�CB� >?�?[��X��ܚX�K??K[XZ[?�?\�\�][��X��ܚX�P?�?\�˘\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[� >?�??K[XZ[?�?\�\�Z?[???�?\�˘\?X�?K�ܙ�B�B
Re: SparkR parallelize not found with 1.4.1?
Thanks! It's good to know --- Original Message --- From: "Eskilson,Aleksander" Sent: June 25, 2015 5:57 AM To: "Felix C" , user@spark.apache.org Subject: Re: SparkR parallelize not found with 1.4.1? Hi there, Parallelize is part of the RDD API which was made private for Spark v. 1.4.0. Some functions in the RDD API were considered too low-level to expose, so only most of the DataFrame API is currently public. The original rationale for this decision can be found on the issue's JIRA [1]. The devs are still considering which parts of the RDD API, if any, should be made public for later releases. If you have some use case that you feel is most easily addressed by the functions currently private in the RDD API, go ahead and let the dev mailing list know. Alek [1] -- https://issues.apache.org/jira/browse/SPARK-7230 On 6/25/15, 12:24 AM, "Felix C" wrote: >Hi, > >It must be something very straightforward... > >Not working: >parallelize(sc) >Error: could not find function "parallelize" > >Working: >df <- createDataFrame(sqlContext, localDF) > >What did I miss? >Thanks >?B�CB� >?�?[��X��ܚX�K??K[XZ[?�?\�\�][��X��ܚX�P?�?\�˘\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[� >?�??K[XZ[?�?\�\�Z?[???�?\�˘\?X�?K�ܙ�B�B CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. B�CB�� [��X��ܚX�KK[XZ[� \�\�][��X��ܚX�P �\�˘\X�K�ܙ�B��܈Y][ۘ[��[X[� �K[XZ[� \�\�Z[ �\�˘\X�K�ܙ�B�B - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SparkR parallelize not found with 1.4.1?
Hi there, Parallelize is part of the RDD API which was made private for Spark v. 1.4.0. Some functions in the RDD API were considered too low-level to expose, so only most of the DataFrame API is currently public. The original rationale for this decision can be found on the issue's JIRA [1]. The devs are still considering which parts of the RDD API, if any, should be made public for later releases. If you have some use case that you feel is most easily addressed by the functions currently private in the RDD API, go ahead and let the dev mailing list know. Alek [1] -- https://issues.apache.org/jira/browse/SPARK-7230 On 6/25/15, 12:24 AM, "Felix C" wrote: >Hi, > >It must be something very straightforward... > >Not working: >parallelize(sc) >Error: could not find function "parallelize" > >Working: >df <- createDataFrame(sqlContext, localDF) > >What did I miss? >Thanks >?B�CB� >?�?[��X��ܚX�K??K[XZ[?�?\�\�][��X��ܚX�P?�?\�˘\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[� >?�??K[XZ[?�?\�\�Z?[???�?\�˘\?X�?K�ܙ�B�B CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
SparkR parallelize not found with 1.4.1?
Hi, It must be something very straightforward... Not working: parallelize(sc) Error: could not find function "parallelize" Working: df <- createDataFrame(sqlContext, localDF) What did I miss? Thanks
