imay commented on issue #1582: Enable Partition Discovery for Broker Load URL: https://github.com/apache/incubator-doris/issues/1582#issuecomment-518493041 > > I think we can support listing path like `"base_dir/*/*/*"` > > It seems a little weird about this syntax. What about support recursively listing files of path(eg, "base_dir/" or "base_dir/*") iff users specify _**columns_from_path**_ by _**[COLUMNS FROM PATH AS (columns_from_path)]**_, which rarely have conflicts with previous usage. If you think wildcard is weird. I think we can keep `DATA INDIR`, and remove the `[PATH START WITH "base_path"]` clause. And if users specify the `columns_from_path` clause we can recursive directory according to it, if they don't we only traverse one depth. For example if user specify `DATA INDIR("/path/to/dir")` and with `COLUMNS FROM PATH AS (k1, k2)`. then we will traverse the two-level directory to get the corresponding import files. We will get file path like "/path/to/dir/k1=1/k2=2/file1". And if users don't specify `COLUMNS FROM PATH` clause, we will only traverse one level directory, and get files like "/path/to/dir/file1". And for `DATA INFILE` with `COLUMNS FROM PATH`, we will try to parse partition columns from user specified file path. And if we can't get match information, we should return error to users.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
