lakhdar adil created SPARK-19015:
------------------------------------
Summary: SQL request with transformation cannot be eecuted if not
run first a scan table
Key: SPARK-19015
URL: https://issues.apache.org/jira/browse/SPARK-19015
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.0
Reporter: lakhdar adil
Hello,
I have a spark streaming wich turn on kafka and send results to ElasticSearch.
I have an union request between two tables: "statswithrowid" table and
"queryes" table
sqlContext.sql(s"select id, rowid,agentId,datecalcul,'KAFKA' as source from
statswithrowid where id IN ($ids) and agentId = '$agent' UNION select id,
rowid,agentId,datecalcul, 'ES' as source from queryes where agentId = '$agent'")
This request cannot be executed lonely. Today i need to execute first those two
requests so that my union request can be working fine. Please find below my two
requests which must be launched first before union request :
request on "statswithrowid " table :
sqlContext.sql(s"select id, rowid,agentId,datecalcul,'KAFKA' as
source from statswithrowid where id IN ($ids) and agentId = '$agent'").show()
request on "queryes" table :
sqlContext.sql(s"select id, rowid,agentId,datecalcul, 'ES' as
source from queryes where agentId = '$agent'").show()
For information : if i don't mention .show() on two requests to launch before
union, nothing can work.
Why i need to launch that first before making union request ? what is the best
way to work with union request ? i try union with dataframe, and i have the
same probleme.
I look forward your reply. Thank you in advance
Best regards,
Adil LAKHDAR
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
