[ https://issues.apache.org/jira/browse/SPARK-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yin Huai updated SPARK-8685: ---------------------------- Description: I have the following code: {code} from pyspark import SQLContext d1 = [{'name':'bob', 'country': 'usa', 'age': 1}, {'name':'alice', 'country': 'jpn', 'age': 2}, {'name':'carol', 'country': 'ire', 'age': 3}] d2 = [{'name':'bob', 'country': 'usa', 'colour':'red'}, {'name':'carol', 'country': 'ire', 'colour':'green'}] r1 = sc.parallelize(d1) r2 = sc.parallelize(d2) sqlContext = SQLContext(sc) df1 = sqlContext.createDataFrame(d1) df2 = sqlContext.createDataFrame(d2) df1.join(df2, df1.name == df2.name and df1.country == df2.country, 'left_outer').collect() {code} When I run it I get the following, (notice in the first row, all join keys are take from the right-side and so are blanked out): {code} [Row(age=2, country=None, name=None, colour=None, country=None, name=None), Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'), Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')] {code} I would expect to get (though ideally without duplicate columns): {code} [Row(age=2, country=u'ire', name=u'alice', colour=None, country=None, name=None), Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'), Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')] {code} The workaround for now is this rather clunky piece of code: {code} df2 = sqlContext.createDataFrame(d2).withColumnRenamed('name', 'name2').withColumnRenamed('country', 'country2') df1.join(df2, df1.name == df2.name2 and df1.country == df2.country2, 'left_outer').collect() {code} was: I have the following code: from pyspark import SQLContext d1 = [{'name':'bob', 'country': 'usa', 'age': 1}, {'name':'alice', 'country': 'jpn', 'age': 2}, {'name':'carol', 'country': 'ire', 'age': 3}] d2 = [{'name':'bob', 'country': 'usa', 'colour':'red'}, {'name':'carol', 'country': 'ire', 'colour':'green'}] r1 = sc.parallelize(d1) r2 = sc.parallelize(d2) sqlContext = SQLContext(sc) df1 = sqlContext.createDataFrame(d1) df2 = sqlContext.createDataFrame(d2) df1.join(df2, df1.name == df2.name and df1.country == df2.country, 'left_outer').collect() When I run it I get the following, (notice in the first row, all join keys are take from the right-side and so are blanked out): [Row(age=2, country=None, name=None, colour=None, country=None, name=None), Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'), Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')] I would expect to get (though ideally without duplicate columns): [Row(age=2, country=u'ire', name=u'alice', colour=None, country=None, name=None), Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'), Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')] The workaround for now is this rather clunky piece of code: df2 = sqlContext.createDataFrame(d2).withColumnRenamed('name', 'name2').withColumnRenamed('country', 'country2') df1.join(df2, df1.name == df2.name2 and df1.country == df2.country2, 'left_outer').collect() > dataframe left joins are not working as expected in pyspark > ----------------------------------------------------------- > > Key: SPARK-8685 > URL: https://issues.apache.org/jira/browse/SPARK-8685 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL > Affects Versions: 1.4.0 > Environment: ubuntu 14.04 > Reporter: axel dahl > > I have the following code: > {code} > from pyspark import SQLContext > d1 = [{'name':'bob', 'country': 'usa', 'age': 1}, > {'name':'alice', 'country': 'jpn', 'age': 2}, > {'name':'carol', 'country': 'ire', 'age': 3}] > d2 = [{'name':'bob', 'country': 'usa', 'colour':'red'}, > {'name':'carol', 'country': 'ire', 'colour':'green'}] > r1 = sc.parallelize(d1) > r2 = sc.parallelize(d2) > sqlContext = SQLContext(sc) > df1 = sqlContext.createDataFrame(d1) > df2 = sqlContext.createDataFrame(d2) > df1.join(df2, df1.name == df2.name and df1.country == df2.country, > 'left_outer').collect() > {code} > When I run it I get the following, (notice in the first row, all join keys > are take from the right-side and so are blanked out): > {code} > [Row(age=2, country=None, name=None, colour=None, country=None, name=None), > Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', > name=u'bob'), > Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', > name=u'alice')] > {code} > I would expect to get (though ideally without duplicate columns): > {code} > [Row(age=2, country=u'ire', name=u'alice', colour=None, country=None, > name=None), > Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', > name=u'bob'), > Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', > name=u'alice')] > {code} > The workaround for now is this rather clunky piece of code: > {code} > df2 = sqlContext.createDataFrame(d2).withColumnRenamed('name', > 'name2').withColumnRenamed('country', 'country2') > df1.join(df2, df1.name == df2.name2 and df1.country == df2.country2, > 'left_outer').collect() > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org