Unsubscribe

2023-07-30 Thread Ali Bajwa
Unsubscribe

Question regarding join with multiple columns with pyspark

2015-04-23 Thread Ali Bajwa
Hi experts, Sorry if this is a n00b question or has already been answered... Am trying to use the data frames API in python to join 2 dataframes with more than 1 column. The example I've seen in the documentation only shows a single column - so I tried this: Example code import pandas a

Re: Question regarding join with multiple columns with pyspark

2015-04-24 Thread Ali Bajwa
Any ideas on this? Any sample code to join 2 data frames on two columns? Thanks Ali On Apr 23, 2015, at 1:05 PM, Ali Bajwa wrote: > Hi experts, > > Sorry if this is a n00b question or has already been answered... > > Am trying to use the data frames API in python to join 2 da

PySpark: slicing issue with dataframes

2015-04-28 Thread Ali Bajwa
li', 'Daniel'], 'Lastname': ['Jones', 'Bajwa', 'Day']}) a = hc.createDataFrame(A) print A b = a.select(a.Firstname[:2]) print b.toPandas() c = a.select(a.Lastname[2:]) print c.toPandas() Output: Firstname Lastname 0 JamesJones 1

Re: Question regarding join with multiple columns with pyspark

2015-04-28 Thread Ali Bajwa
y=u'1993~1', mn=1, price=100, yr=1993, joiningKey=u'1993~1', > mn=1, name=u'A', yr=1993) > > - > > Field concat method, works as well.... > > YMJNA = YM1DF.join(YM2DF,YM1DF.joiningKey==YM2DF

Re: PySpark: slicing issue with dataframes

2015-05-03 Thread Ali Bajwa
Friendly reminder on this one. Just wanted to get a confirmation that this is not by design before I logged a JIRA Thanks! Ali On Tue, Apr 28, 2015 at 9:53 AM, Ali Bajwa wrote: > Hi experts, > > Trying to use the "slicing" functionality in strings as part of a Spark > p