Re: Accessing Hive Tables in Spark

Hi Tushar, It seems Spark is not able to access the metastore. It may be because you are using derby metastases which is maintained locally. Please check all your configurations and that Spark has access to the hive-site.xml file with the metastore uri. Thanks, Marco On Tue, 10 Apr 2018, 08:20 T

Accessing Hive Tables in Spark

Hi Everyone, I was accessing Hive Tables in Spark SQL using Scala submitted by spark-submit command. When I ran in cluster mode then got error like : Table not found But the same is working while submitted as client mode. Please help me to understand why? Distribution : Hortonworks Thanks in adv

pyspark.daemon exhaust a lot of memory

Hi All, We are running spark 2.1.1 on Hadoop YARN 2.6.5. We found the pyspark.daemon process consume more than 300GB memory. However, according to https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals, the daemon process shouldn't have this problem. Also, we find the daemon proces

Correlated subqueries in the DataFrame API

I just submitted SPARK-23945 but wanted to double check here to make sure I didn't miss something fundamental. Correlated subqueries are tracked at a high level in SPARK-18455 , but it's not clea

Re: Fair scheduler pool leak

If I understand what you're trying to do correctly, I think you really just want one pool, but you want to change the mode *within* the pool to be FAIR as well https://spark.apache.org/docs/latest/job-scheduling.html#configuring-pool-properties you'd still need to change the conf file to set up t

Re: Clarify window behavior in Spark SQL

Hi Li, You might find my pending PR useful: https://github.com/apache/spark/pull/20045/files It contains a big bunch of test cases covering the windowing functionality, showing and checking the behavior of a number of special cases. On Wed, Apr 4, 2018 at 4:26 AM, Reynold Xin wrote: > Thanks Li