Question regarding Projection PushDown

2021-08-27 Thread satyajit vegesna
Hi All, Please help with below question, I am trying to build my own data source to connect to CustomAerospike. Now I am almost done with everything, but still not sure how to implement Projection Pushdown while selecting nested columns. Spark does implicit for column projection pushdown, but

Re: [VOTE] Release Spark 3.2.0 (RC1)

2021-08-27 Thread Sean Owen
Maybe, I'm just confused why it's needed at all. Other profiles that add a dependency seem OK, but something's different here. One thing we can/should change is to simply remove the block in the profile. It should always be a direct dep in Scala 2.13 (which lets us take out the profiles in

Re: How to improve the concurrent query performance of spark SQL query

2021-08-27 Thread Mich Talebzadeh
There are many ways of interacting with Hive DW from Spark. You can either use the API from Spark to Hive native or you can use JDBC connection (local or remote spark). What is the reference to the driver in this context? Bottom line using concurrent queries, you will have to go through Hive and