yuanlihan commented on issue #1569: Enable Partition Discovery When Loading Data from Parquet File URL: https://github.com/apache/incubator-doris/pull/1569#issuecomment-517104868 > @yuanlihan > > Firstly, thanks for your improvement. I have some questions about this PR. > > 1. Should we add an option to switch this feature off? Because this will fill partition row automatically which should be known by users. > 2. Can this patch work with other format of file? like CSV format? And how can user use this function. > 3. How this function work with other load functions, like user specified columns and set operation. For example, if the partition column name is different in Doris table and Spark. @imay Thanks for your review 1. In my opinion, it is quite intuitive to enable partition discovery as well as recursively list files by default. We extract partition columns iff them was defined/needed in the specified table. 2. We can support partition discovery of other file sources(including Text/CSV/JSON/ORC/Parquet) like in Spark. May be users will prefer having this feature by default, which rarely have conflicts with their previous usage. 3. I will dig more about this and try to solve it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
