cxzl25 commented on pull request #32583: URL: https://github.com/apache/spark/pull/32583#issuecomment-843285679
In our production environment, there is a partition table with about 80,000 partitions, and dt is the partition field. In this case, Spark pulls all the partitions, which generates a lot of SQL for querying partitions, which puts a lot of pressure on the MetaStore Server, and the speed is more than 100 times slower than Hive. ```sql select a.* from X a where substr(a.dt,1,10) = '2018-01-07' limit 10; ``` Hive: Time taken: 2.816 seconds, Fetched: 10 row(s) Spark: Time taken: 248 seconds, Fetched: 10 row(s)   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
