Re: Add python library with native code

ack is a library on top of either pex or conda-pack to make your Python code easily available on a cluster. Masood __ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 6/5/20 4:29 AM, Stone Zhong wrote: Thanks Dark. Looked at tha

Re: spark-submit exit status on k8s

, then the Spark app must have failed. The above should work if you want to avoid dealing with the REST API for monitoring. Masood __ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 4/4/20 10:54 AM, Masood Krohy wrote: I'm n

Re: spark-submit exit status on k8s

. Masood ______ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 4/4/20 7:49 AM, Marshall Markham wrote: Thank you very much Masood for your fast response. Last question, is the current status in Jira representative of the status of the ticket

Re: spark-submit exit status on k8s

in a loop to the REST API with few seconds delay between each call while the execution is in progress until the application fails or succeeds. https://spark.apache.org/docs/latest/monitoring.html#rest-api Hope this helps. Masood ______ Masood Krohy, Ph.D. Data Science Advisor|Pla

Re: [Pyspark 2.3+] Timeseries with Spark

llel job (a.k.a, embarrassingly parallel). I gave a talk on this which you may find useful: https://www.analytical.works/Talk-spark-ml.html Masood ______ Masood Krohy, Ph.D. Data Science Advisor|Platform Architect https://www.analytical.works On 12/29/19 11:30 AM, Rishi Shah wrote: Hi All,

RE: build models in parallel

ch. Masood ------ Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Xiaomeng Wan A : User Date : 2016-11-29 11:54 Objet : build models in parallel I want to divide big data into groups (eg groupby

Re: Cluster deploy mode driver location

e) Masood ------ Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Silvio Fiorito A : "saif.a.ell...@wellsfargo.com" , "user@spark.apache.org" Date : 2016-11-22 08:02 Objet : Re: C

Re: LinearRegressionWithSGD and Rank Features By Importance

ined your model. This is now becoming more of a Data Science/ML problem and not a Spark issue and is probably best kept off this list. Do some reading on the topic and get back to me direct; I'll respond when possible. Hope this has helped. Masood ---------- Masoo

Re: Live data visualisations with Spark

+1 for Zeppelin. See https://community.hortonworks.com/articles/10365/apache-zeppelin-and-sparkr.html -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Vadim Semenov A : An

Re: LinearRegressionWithSGD and Rank Features By Importance

caled features, so you gain if you scale the features beforehand. Masood -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporation http://ca.linkedin.com/in/masoodkh De :Carlo.Allocca A : Mohit Jaggi Cc :Carlo.Allocc

Re: Deep learning libraries for scala

marks last time I checked. Alexander Ulanov of HP made a presentation of the options few months ago: https://www.oreilly.com/learning/distributed-deep-learning-on-spark Masood -- Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporati

Re: Getting the IP address of Spark Driver in yarn-cluster mode

st_length = len('worker1_hostname') amHost = html[amHost_start : amHost_start + amHost_length] print 'amHostHttpAddress is: ' + amHost try: self.websock = ... print 'Connected to server running on %s' % nodes[amHost] except: print '

Getting the IP address of Spark Driver in yarn-cluster mode

] ) # output in YARN stdout log: SparkJobInfo(jobId=4, stageIds=JavaObject id=o101, status='SUCCEEDED') ------ Masood Krohy, Ph.D. Data Scientist, Intact Lab-R&D Intact Financial Corporation