[jira] [Commented] (SPARK-17538) sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0
[ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196236#comment-16196236 ] Hyukjin Kwon commented on SPARK-17538: -- I can't reproduce this. Would you be able to provide more steps to reproduce this? For example, how the initial dataframe was created? > sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0 > - > > Key: SPARK-17538 > URL: https://issues.apache.org/jira/browse/SPARK-17538 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: os - linux > cluster -> yarn and local >Reporter: Srinivas Rishindra Pothireddi > > I have a production job in spark 1.6.2 that registers several dataframes as > tables. > After testing the job in spark 2.0.0, I found that one of the dataframes is > not getting registered as a table. > Line 353 of my code --> self.sqlContext.registerDataFrameAsTable(anonymousDF, > "anonymousTable") > line 354 of my code --> df = self.sqlContext.sql("select AnonymousFiled1, > AnonymousUDF( AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > my stacktrace > File "anonymousFile.py", line 354, in anonymousMethod > df = self.sqlContext.sql("select AnonymousFiled1, AnonymousUDF( > AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/context.py", > line 350, in sql > return self.sparkSession.sql(sqlQuery) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/session.py", > line 541, in sql > return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", > line 933, in __call__ > answer, self.gateway_client, self.target_id, self.name) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/utils.py", > line 69, in deco > raise AnalysisException(s.split(': ', 1)[1], stackTrace) > AnalysisException: u'Table or view not found: anonymousTable; line 1 pos 61' > The same code is working perfectly fine in spark-1.6.2 > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17538) sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0
[ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493075#comment-15493075 ] Srinivas Rishindra Pothireddi commented on SPARK-17538: --- Hi [~srowen], I updated the description of the jira as per your suggestion. Can you please let me know if you need anymore information > sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0 > - > > Key: SPARK-17538 > URL: https://issues.apache.org/jira/browse/SPARK-17538 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: os - linux > cluster -> yarn and local >Reporter: Srinivas Rishindra Pothireddi > > I have a production job in spark 1.6.2 that registers several dataframes as > tables. > After testing the job in spark 2.0.0, I found that one of the dataframes is > not getting registered as a table. > Line 353 of my code --> self.sqlContext.registerDataFrameAsTable(anonymousDF, > "anonymousTable") > line 354 of my code --> df = self.sqlContext.sql("select AnonymousFiled1, > AnonymousUDF( AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > my stacktrace > File "anonymousFile.py", line 354, in anonymousMethod > df = self.sqlContext.sql("select AnonymousFiled1, AnonymousUDF( > AnonymousFiled1 ) as AnonymousFiled3 from anonymousTable") > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/context.py", > line 350, in sql > return self.sparkSession.sql(sqlQuery) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/session.py", > line 541, in sql > return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", > line 933, in __call__ > answer, self.gateway_client, self.target_id, self.name) > File > "/home/anonymousUser/Downloads/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/utils.py", > line 69, in deco > raise AnalysisException(s.split(': ', 1)[1], stackTrace) > AnalysisException: u'Table or view not found: anonymousTable; line 1 pos 61' > The same code is working perfectly fine in spark-1.6.2 > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17538) sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0
[ https://issues.apache.org/jira/browse/SPARK-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492937#comment-15492937 ] Srinivas Rishindra Pothireddi commented on SPARK-17538: --- Hi [~srowen], I will fix this as soon as possible as you suggested. > sqlContext.registerDataFrameAsTable is not working sometimes in pyspark 2.0.0 > - > > Key: SPARK-17538 > URL: https://issues.apache.org/jira/browse/SPARK-17538 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: os - linux > cluster -> yarn and local >Reporter: Srinivas Rishindra Pothireddi > > I have a production job in spark 1.6.2 that registers four dataframes as > tables. After testing the job in spark 2.0.0 one of the dataframes is not > getting registered as a table. > output of sqlContext.tableNames() just after registering the fourth dataframe > in spark 1.6.2 is > temp1,temp2,temp3,temp4 > output of sqlContext.tableNames() just after registering the fourth dataframe > in spark 2.0.0 is > temp1,temp2,temp3 > so when the table 'temp4' is used by the job at a later stage an > AnalysisException is raised in spark 2.0.0 > There are no changes in the code whatsoever. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org