Re: SPARK_MASTER_IP actually expects a DNS name, not IP address

[Robert Dodier]

Nicholas,

FWIW the --ip option seems to have been deprecated in commit d90d2af1,
but that was a pretty big commit, lots of other stuff changed, and there
isn't any hint in the log message as to the reason for changing --ip.

best,

Robert Dodier

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: PMML export for LinearRegressionModel

[Fazlan Nazeem]

Hi Joseph,

That's great. Also It would be great if spark extends the PMML support to
models which are not PMML supported right now.
e.g

   - Decision Tree
   - Random Forest
   - Naive Bayes


On Sun, Oct 18, 2015 at 2:55 AM, Joseph Bradley 
wrote:

> Thanks for bringing this up!  We need to add PMML export methods to the
> spark.ml API.  I just made a JIRA for tracking that:
> https://issues.apache.org/jira/browse/SPARK-11171
>
> Joseph
>
> On Thu, Oct 15, 2015 at 2:58 AM, Fazlan Nazeem  wrote:
>
>> Ok It turns out I was using the wrong LinearRegressionModel which was in  
>> package
>> org.apache.spark.ml.regression;.
>>
>>
>>
>> On Thu, Oct 15, 2015 at 3:23 PM, Fazlan Nazeem  wrote:
>>
>>> This is the API doc for LinearRegressionModel. It does not implement
>>> PMMLExportable
>>>
>>> https://spark.apache.org/docs/latest/api/java/index.html
>>>
>>> On Thu, Oct 15, 2015 at 3:11 PM, canan chen  wrote:
>>>
 The method toPMML is in trait PMMLExportable

 *LinearRegressionModel has this trait, you should be able to call *
 *LinearRegressionModel#toPMML*

 On Thu, Oct 15, 2015 at 5:25 PM, Fazlan Nazeem 
 wrote:

> Hi
>
> I am trying to export a LinearRegressionModel in PMML format.
> According to the following resource[1] PMML export is supported for
> LinearRegressionModel.
>
> [1] https://spark.apache.org/docs/latest/mllib-pmml-model-export.html
>
> But there is *no* *toPMML* method in *LinearRegressionModel* class
> although LogisticRegressionModel, ReidgeRegressionModel,SVMModel etc has
> toPMML method.
>
> Can someone explain what is the issue here?
>
> --
> Thanks & Regards,
>
> Fazlan Nazeem
>
> *Software Engineer*
>
> *WSO2 Inc*
> Mobile : +94772338839
> <%2B94%20%280%29%20773%20451194>
> fazl...@wso2.com
>


>>>
>>>
>>> --
>>> Thanks & Regards,
>>>
>>> Fazlan Nazeem
>>>
>>> *Software Engineer*
>>>
>>> *WSO2 Inc*
>>> Mobile : +94772338839
>>> <%2B94%20%280%29%20773%20451194>
>>> fazl...@wso2.com
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>>
>> Fazlan Nazeem
>>
>> *Software Engineer*
>>
>> *WSO2 Inc*
>> Mobile : +94772338839
>> <%2B94%20%280%29%20773%20451194>
>> fazl...@wso2.com
>>
>
>


-- 
Thanks & Regards,

Fazlan Nazeem

*Software Engineer*

*WSO2 Inc*
Mobile : +94772338839
<%2B94%20%280%29%20773%20451194>
fazl...@wso2.com

test failed due to OOME

[Ted Yu]

From
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/3846/console
:

SparkListenerSuite:- basic creation and shutdown of LiveListenerBus-
bus.stop() waits for the event queue to completely drain- basic
creation of StageInfo- basic creation of StageInfo with shuffle-
StageInfo with fewer tasks than partitions- local metrics-
onTaskGettingResult() called when result fetched remotely *** FAILED
***  org.apache.spark.SparkException: Job aborted due to stage
failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost
task 0.0 in stage 0.0 (TID 0, localhost): java.lang.OutOfMemoryError:
Java heap space at java.util.Arrays.copyOf(Arrays.java:2271)at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)  at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) 
at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at
java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1852)
at
java.io.ObjectOutputStream.write(ObjectOutputStream.java:708)   at
org.apache.spark.util.Utils$.writeByteBuffer(Utils.scala:182)   at
org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply$mcV$sp(TaskResult.scala:52)
  at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1160) at
org.apache.spark.scheduler.DirectTaskResult.writeExternal(TaskResult.scala:49)  
at
java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458)  
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)   at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
  at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:256)   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at
java.lang.Thread.run(Thread.java:745)


Should more heap be given to test suite ?


Cheers