abyssnlp commented on issue #108: URL: https://github.com/apache/hudi-rs/issues/108#issuecomment-2271167479
Hi @aditya-raval-genea, Thanks for sharing the details. I created an isolated environment with the configuration you'd shared (Python `3.12` and all the relevant packages with pinned versions), and I'm **able to successfully read** from the Hudi table. I simulated a dataset with around 45k records on s3 using Hudi version `0.15.0`. Here are my 2 cents: - It might be that the partitioning scheme on `customer_uuid` is not appropriate as partitioning should be done on low cardinality columns. If the cardinality is too high, you would have too many small files and there would be a huge file IO overhead. Try partitioning with the `year/month/day` scheme. - I noticed you have pyspark as one of your dependencies. In case you don't manage to get `hudi-rs` to work for your use-case, it might also be feasible to use pyspark to read the Hudi table as it has excellent integration with it. You can read more about it [here](https://hudi.apache.org/docs/quick-start-guide/#spark-shellsql). Let me know if either of these work out for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
