I agree with Wim's assessment of data engineering / ETL vs Data Science. I wrote pipelines/frameworks for large companies and scala was a much better choice. But for ad-hoc work interfacing directly with data science experiments pyspark presents less friction.
On Sat, 10 Oct 2020 at 13:03, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Many thanks everyone for their valuable contribution. > > We all started with Spark a few years ago where Scala was the talk of the > town. I agree with the note that as long as Spark stayed nish and elite, > then someone with Scala knowledge was attracting premiums. In fairness in > 2014-2015, there was not much talk of Data Science input (I may be wrong). > But the world has moved on so to speak. Python itself has been around > a long time (long being relative here). Most people either knew UNIX Shell, > C, Python or Perl or a combination of all these. I recall we had a director > a few years ago who asked our Hadoop admin for root password to log in to > the edge node. Later he became head of machine learning somewhere else and > he loved C and Python. So Python was a gift in disguise. I think Python > appeals to those who are very familiar with CLI and shell programming (Not > GUI fan). As some members alluded to there are more people around with > Python knowledge. Most managers choose Python as the unifying development > tool because they feel comfortable with it. Frankly I have not seen a > manager who feels at home with Scala. So in summary it is a bit > disappointing to abandon Scala and switch to Python just for the sake of it. > > Disclaimer: These are opinions and not facts so to speak :) > > Cheers, > > > Mich > > > > > > > On Fri, 9 Oct 2020 at 21:56, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> I have come across occasions when the teams use Python with Spark for >> ETL, for example processing data from S3 buckets into Snowflake with Spark. >> >> The only reason I think they are choosing Python as opposed to Scala is >> because they are more familiar with Python. Since Spark is written in >> Scala, itself is an indication of why I think Scala has an edge. >> >> I have not done one to one comparison of Spark with Scala vs Spark with >> Python. I understand for data science purposes most libraries like >> TensorFlow etc. are written in Python but I am at loss to understand the >> validity of using Python with Spark for ETL purposes. >> >> These are my understanding but they are not facts so I would like to get >> some informed views on this if I can? >> >> Many thanks, >> >> Mich >> >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >
