Why don't you do a normal .saveAsTextFiles? Thanks Best Regards
On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <[email protected]> wrote: > Thanx for reply !! > > YES , Either it should write on any machine of cluster or Can you please > help me ... that how to do this . Previously i was using writing using > collect () , so some of my tuples are missing while writing. > > //previous logic that was just creating the file on master - > > newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() { > > @Override > public Void call(JavaRDD<String> v1, Time v2) throws Exception { > for(String s:v1.collect()) { > // System.out.println("v1 here is " + v1 + "-------" + s); > spoutlog.batchLogwriter(System.currentTimeMillis(), > "spout-MSGID," + msgeditor.getMessageId(s)); > // System.out.println(msgeditor.getMessageId(s)); > } > return null; > } > }); > > > > > On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher < > [email protected]> wrote: > >> Is spoutLog just a non-spark file writer? If you run that in the map call >> on a cluster its going to be writing in the filesystem of the executor its >> being run on. I'm not sure if that's what you intended. >> >> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <[email protected]> >> wrote: >> >>> Running perfectly in local system but not writing to file in cluster mode >>> .ANY suggestions please .. >>> >>> >>> //msgid is long counter >>> >>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, >>> String>() { >>> @Override >>> public String call(String v1) throws Exception { >>> String s1=msgId+"@"+v1; >>> System.out.println(s1); >>> msgId++; >>> try { >>> *//filewriter logic >>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + >>> msgeditor.getMessageId(s1));* >>> } catch (Exception e) { >>> >>> System.out.println("exeception is here"); >>> e.printStackTrace(); >>> throw e; >>> } >>> System.out.println("msgid,"+msgId); >>> return msgeditor.addMessageId(v1,msgId); >>> } >>> }); >>> >>> >>> -- >>> Thanks & Regards, >>> Anshu Shukla >>> >>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <[email protected]> >>> wrote: >>> >>>> Can not we write some data to a txt file in parallel with multiple >>>> executors running in parallel ?? >>>> >>>> >>>> -- >>>> Thanks & Regards, >>>> Anshu Shukla >>>> >>> >>> >>> >>> -- >>> Thanks & Regards, >>> Anshu Shukla >>> >> >> > > > -- > Thanks & Regards, > Anshu Shukla >
