ziudu commented on issue #11438: URL: https://github.com/apache/hudi/issues/11438#issuecomment-2162613994
with a broadcast variable in Method 2 there is no Spark SQL planning issue  Method 1 is definitely not good. Even with 20K values the planning takes 1.5 minutes. 1 million would even kill Catalyst Optimizer.  I'm looking for a way to select millions of records from a very large table (billions of records) via Record Index. RLI's performance is amazing for thousands of records, so I want to do a pressure testing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
