Re: Add python library with native code

is a library on top of either pex or conda-pack to make your Python code easily available on a cluster. Masood ______ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 6/5/20 4:29 AM, Stone Zhong wrote: Thanks Dark. Looked at that arti

Re: spark-submit exit status on k8s

, then the Spark app must have failed. The above should work if you want to avoid dealing with the REST API for monitoring. Masood __ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 4/4/20 10:54 AM, Masood Krohy wrote: I'm

Re: spark-submit exit status on k8s

__ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 4/4/20 7:49 AM, Marshall Markham wrote: Thank you very much Masood for your fast response. Last question, is the current status in Jira representative of the status of the ticket within

Re: spark-submit exit status on k8s

in a loop to the REST API with few seconds delay between each call while the execution is in progress until the application fails or succeeds. https://spark.apache.org/docs/latest/monitoring.html#rest-api Hope this helps. Masood __ Masood Krohy, Ph.D. Data Science Advisor|Platform

Re: [Pyspark 2.3+] Timeseries with Spark

, embarrassingly parallel). I gave a talk on this which you may find useful: https://www.analytical.works/Talk-spark-ml.html Masood __ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 12/29/19 11:30 AM, Rishi Shah wrote: Hi All, Checking

RE: build models in parallel

-- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Xiaomeng Wan <shawn...@gmail.com> A : User <user@spark.apache.org> Date : 2016-11-29 11:54 Objet : build models in parallel I want to div

Re: Cluster deploy mode driver location

e) Masood ------ Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Silvio Fiorito <silvio.fior...@granturing.com> A : "saif.a.ell...@wellsfargo.com" <saif.a.ell...@wellsfargo.com>

Re: LinearRegressionWithSGD and Rank Features By Importance

. This is now becoming more of a Data Science/ML problem and not a Spark issue and is probably best kept off this list. Do some reading on the topic and get back to me direct; I'll respond when possible. Hope this has helped. Masood -- Masood Krohy, Ph.D. Data

Re: Live data visualisations with Spark

+1 for Zeppelin. See https://community.hortonworks.com/articles/10365/apache-zeppelin-and-sparkr.html -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Vadim Semenov <vadim.s

Re: LinearRegressionWithSGD and Rank Features By Importance

features, so you gain if you scale the features beforehand. Masood -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Carlo.Allocca <carlo.allo...@open.ac.uk> A : Mohit Jaggi &l

Re: Deep learning libraries for scala

last time I checked. Alexander Ulanov of HP made a presentation of the options few months ago: https://www.oreilly.com/learning/distributed-deep-learning-on-spark Masood -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation De

Re: Getting the IP address of Spark Driver in yarn-cluster mode

lf.websock = ... print 'Connected to server running on %s' % nodes[amHost] except: print 'Could not connect to server on %s' % nodes[amHost] ------ Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation De :Steve Lou

Getting the IP address of Spark Driver in yarn-cluster mode

] ) # output in YARN stdout log: SparkJobInfo(jobId=4, stageIds=JavaObject id=o101, status='SUCCEEDED') -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R Intact Financial Corporation