Re: About HostName display in SparkUI
In the conf/slaves file, are you having the ip addresses? or the hostnames? Thanks Best Regards On Sat, Jun 13, 2015 at 9:51 PM, Sea 261810...@qq.com wrote: In spark 1.4.0, I find that the Address is ip (it was hostname in v1.3.0), why? who did it?
Using queueStream
JavaDStreamString inputStream = ssc.queueStream(rddQueue); Can this rddQueue be of dynamic type in nature .If yes then how to make it run untill rddQueue is not finished . Any other way to get rddQueue from a dynamically updatable Normal Queue . -- Thanks Regards, SERC-IISC Anshu Shukla
Random Forest driver memory
hello, i have noticed that the random forest implementation crashes when to many trees/ to big maxDepth is used. im guessing that this is something to do with the amount of nodes that need to be kept in driver's memory during the run. but when i examined the nodes structure is seems rather small does anyone now where does the memory issue come from? thanks, Isca
Re?? About HostName display in SparkUI
In the conf/slaves file, I have hostnames. Before 1.4.0, it is okay. I view the code in class org.apache.spark.util.Utils, I alter function localHostName and localHostNameForURI, and it turns back to hostnames again. I just don't know why to change these basic functions. Hostname is nice. -- -- ??: Akhil Das;ak...@sigmoidanalytics.com; : 2015??6??15??(??) 5:36 ??: Sea261810...@qq.com; : devdev@spark.apache.org; : Re: About HostName display in SparkUI In the conf/slaves file, are you having the ip addresses? or the hostnames? ThanksBest Regards On Sat, Jun 13, 2015 at 9:51 PM, Sea 261810...@qq.com wrote: In spark 1.4.0, I find that the Address is ip (it was hostname in v1.3.0), why? who did it? 7FAB9BA9@AFBE9573.34FC7E55 Description: Binary data
Problem: Custom Receiver for getting events from a Dynamic Queue
I have written a custom receiver for converting the tuples in the Dynamic
Queue/EventGen to the Dstream.But i dont know why It is only processing
data for some time (3-4 sec.) only and then shows Queue as Empty .ANy
suggestions please ..
--code //
public class JavaCustomReceiver extends ReceiverString implements
ISyntheticEventGen {
EventGen eventGen;
BlockingQueueListString eventQueue;
String csvFileName;
String outSpoutCSVLogFileName;
double scalingFactor;
public JavaCustomReceiver(String csvFileName, String
outSpoutCSVLogFileName, double scalingFactor) {
super(StorageLevel.MEMORY_AND_DISK());
this.csvFileName = csvFileName;
this.outSpoutCSVLogFileName = outSpoutCSVLogFileName;
this.scalingFactor = scalingFactor;
this.eventGen = new EventGen(this,this.scalingFactor);
this.eventGen.launch(this.csvFileName,
this.outSpoutCSVLogFileName); //Launch threads
this.eventQueue = new LinkedBlockingQueueListString();
System.out.println(for watching queue);
}
public void onStart() {
// Start the thread that receives data over a connection
new Thread() {
@Override public void run() {
receive();
}
}.start();
}
public void onStop() {
// There is nothing much to do as the thread calling receive()
// is designed to stop by itself isStopped() returns false
}
/** Create a socket connection and receive data until receiver is stopped */
private void receive() {
try {
// connect to the server
//socket = new Socket(host, port);
//BufferedReader reader = new BufferedReader(new
InputStreamReader(socket.getInputStream()));
// Until stopped or connection broken continue reading
while (!isStopped() ) {
ListString entry = this.eventQueue.take();
String str=;
for(String s:entry)
str+=s+,;
System.out.println(Received data ' + str + ');
store(str);
}
// Restart in an attempt to connect again when server is
active again
restart(Trying to connect again);
}
catch(Throwable t) {
// restart if there is any other error
restart(Error receiving data, t);
}
}
@Override
public StorageLevel storageLevel() {
return StorageLevel.MEMORY_AND_DISK();
}
@Override
public void receive(ListString event) {
// TODO Auto-generated method stub
//System.out.println(Called IN SPOUT### );
try {
this.eventQueue.put(event);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
--
Thanks Regards,
Anshu Shukla
[SparkStreaming] NPE in DStreamCheckPointData.scala:125
I use the attached program to test checkpoint. It's quite simple. When I run the program second time, it will load checkpoint data, that's expected, however I see NPE in driver log. Do you have any idea about the issue? I'm on Spark 1.4.0, thank you very much! == logs == 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream: Restoring checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435313 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435314 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435315 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435316 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435317 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435318 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream$FileInputDStreamCheckpointData: Restoring files for time 143435319 ms - [] 15/06/15 15:27:17 [THREAD ID=main] INFO FileInputDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO MappedDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO MappedDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO ForEachDStream: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] INFO DStreamGraph: Restored checkpoint data 15/06/15 15:27:17 [THREAD ID=main] ERROR StreamingContext: Error starting the context, marking it as stopped java.io.IOException: java.lang.NullPointerException at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1242) at org.apache.spark.streaming.dstream.DStreamCheckpointData.writeObject(DSt reamCheckpointData.scala:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:44 0) at org.apache.spark.streaming.dstream.DStream$$anonfun$writeObject$1.apply$ mcV$sp(DStream.scala:498) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239) at org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:493 ) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1 431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:15 47) at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:44 0) at org.apache.spark.streaming.DStreamGraph$$anonfun$writeObject$1.apply$mcV $sp(DStreamGraph.scala:181) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239) at org.apache.spark.streaming.DStreamGraph.writeObject(DStreamGraph.scala:1 76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:57) at
