gt; rdd2.values.count() * x) is invalid because the values
transformation and count action cannot be performed inside of the rdd1.map
transformation. For more information, see SPARK-5063.
any advices on this? Thanks.
--------------------
Thanks&Best regards!
San.Luo
- 原始邮件
compare.sh to compare chr1.txt and samplechr1.txt, get a
result1.txt. And looping it from 1 to 21 so that those 42 file are compared and
I can get 21 files like result1.txt,result2.txt...result21.txt.
Sorry for not adding some comments for my code.
--------------------
Thanks&Best r
(runmodifyshell)
> val pipeModify = runmodifyshellRDD.pipe("sh
> /opt/data/shellcompare/modify.sh")
>
> pipeModify.collect()
>
>
>
> //running on driver manager
> val shellcompare = List("run","sort.sh")
> val shellcomp
mplechr1.txt, get a
result1.txt. And looping it from 1 to 21 so that those 42 file are compared and
I can get 21 files like result1.txt,result2.txt...result21.txt.
Sorry for not adding some comments for my code.
----------------
Thanks&Best regards!
San.Luo
- 原始邮件 -
RDD(shellcompare)
> val result = List("aggregate","result")
> val resultRDD = sc.makeRDD(result)
> for(j <- 1 to 21){
> shellcompareRDD.pipe("sh /opt/sh/bin/sort.sh
> /opt/data/shellcompare/chr" + j + ".txt /opt/data/shellcompare
sh/bin/sort.sh /opt/data/shellcompare/chr"
+ j + ".txt /opt/data/shellcompare/samplechr" + j + ".txt
/opt/data/shellcompare/result" + j + ".txt 600").collect()
if (j > 1) resultRDD.pipe("cat result" + j + ".txt >>
result1.txt").coll
your code is a big help to me,'cause perl script is the exactly
> thing i wanna try to run in spark. I will have a try.
>
>
>
>
> Thanks&Best regards!
> San.Luo
>
> - 原始邮件 -----
> 发件人：Akhil Das
> 收件人：罗辉
&g
Thanks Akhil,
your code is a big help to me,'cause perl script is the exactly
thing i wanna try to run in spark. I will have a try.
Thanks&Best regards!
San.Luo
- 原始邮件 -
发件人：Akhil Das
收件人：罗辉
抄送人：user
主题：Re: how to distribu
You mean you want to execute some shell commands from spark? Here's
something i tried a while back. https://github.com/akhld/spark-exploit
Thanks
Best Regards
On Sun, May 24, 2015 at 4:53 PM, wrote:
> hello there
>
> I am trying to run a app in which part of it needs to run a
> shell.how
