Re: How to control saveAsTable() warehouse path?

2016-12-15 Thread epettijohn
I don't profess to be an expert on this, but I did face the same problem. A couple of possibilities: 1. If your default Hive database is stored in "/tmp/hive/warehouse/...", then that could be the issue. I recommend creating a database on s3a and then storing the table there (

Parallel read from OracleDB slow, fails on large tables

2016-12-13 Thread epettijohn
I'm running the following code in an attempt to import some tables from our Oracle DB into Spark (2.0.2), and then save them as Parquet tables in S3 (using S3A). The code runs, and does create query-able tables in our Hive Metastore, but it only creates one connection to Oracle (I was expecting