prashantwason commented on pull request #2064: URL: https://github.com/apache/hudi/pull/2064#issuecomment-698099305
@umehrot2 Directly using hudi datasource or delta streamer for testing should work too. I haven't testing this yet so please let me know if that doesn't work. Query side changes are not implemented yet. So this is ingestion side improvement as of today. On Wed, Sep 23, 2020 at 7:09 PM Udit Mehrotra <[email protected]> wrote: > @umehrot2 <https://github.com/umehrot2> I have updated the RFC doc with details > on how to test RFC-15 > <https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements#RFC15:HUDIFileListingandQueryPlanningImprovements-TestPlan>. > Please take a look and let me know if I can help in any way. > > @prashantwason <https://github.com/prashantwason> I missed your earlier > pings on this PR. We will start the testing of this PR with S3. Looking at > the testing details you provided, I am a bit confused, and have a couple of > questions: > > - > > Do we need to directly use HoodieWriteClient from spark-shell to be > able to test this ? Can't we directly use hudi datasource or delta > streamer for testing with following options set to true while writing: > hoodie.metadata.file.listings.enable, > hoodie.metadata.file.listings.verify ? > - > > Also for testing query performance using spark datasource, spark-sql, > hive and presto I am assuming it will detect that metadata table is > present and automatically use that for getting the list ? > > Will review the PR this week as well to understand the implementation > details. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/hudi/pull/2064#issuecomment-698068512>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AN55SSZN3JHLGZ77YHS3RQDSHKS4NANCNFSM4QVMIMDQ> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
