Hi Fokko,
looks like you've fixed the issue that was causing it :)
[AIRFLOW-1562] Spark-sql logging contains deadlock
This is exactly what I was seeing - the process would just freeze on the
second query I guess waiting for the lock on the log file
Thanks!
On Sat, Oct 14, 2017 at 5:07 AM,
Hi Boris,
Interesting. Multiple queries is supported by the spark-sql operator and
this should work using Airflow. Executing SQL from a file:
Fokkos-MBP:~ fokkodriesprong$ spark-sql --driver-java-options
"-Dlog4j.configuration=file:///tmp/log4j.properties" -f query.sql
1
Time taken: 1.976
hi guys,
tried spark_sql_hook to run a multi-statement query (two queries separated
by semi-column ) and it hangs forever. If i comment out the second query,
it runs fine.
Anyone had the same issue? i do not see anything in the code preventing
more one statement.
sql = """
select * from
