yuanlihan opened a new issue #1582: Enable Partition Discovery for Broker Load URL: https://github.com/apache/incubator-doris/issues/1582 **Can not parse partitioned columns and can not recursively list files when using Broker Load** When users try to load data from hdfs files written by Spark jobs, they usually need to extract partitioned columns which are specified by write options of Spark API. And recursively listing files of a path(dir) is needed because there are usually multi partitioned columns. Ex: We try to load data from source: - **base path**: hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/ - **partitioned columns need to be extracted**: city and utc_date - **input path(dir)**: hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/city=beijing/utc_date=2019-06-26 - **detail files**: [hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/city=beijing/utc_date=2019-06-26/0000.csv, hdfs://hdfs_host:hdfs_port/user/palo/data/input/dir/city=beijing/utc_date=2019-06-26/0001.csv, ...] **Expected Interface of Broker Load for Partition Discovery Feature** ``` DATA INDIR ( "path"[, path1, ...] ) [NEGATIVE] INTO TABLE `table_name` [PARTITION (p1, p2)] [COLUMNS TERMINATED BY "column_separator"] [FORMAT AS "file_type"] [BASE_PATH AS "base_path"] [PARTITIONED_COLUMNS AS (partitioned_column_list)] [(column_list)] [SET (k1 = func(k2))] ``` - **path**: path of file or dir. if it is a path of dir then we will recursively list all files under this path - **base_path**: the base path property for partition discovery like in Spark - **partitioned_column_list**: the partitioned columns needed to be extracted from file path **Alternative Interface** ``` DATA INFILE ( "file_path1"[, file_path2, ...] ) [NEGATIVE] INTO TABLE `table_name` [PARTITION (p1, p2)] [COLUMNS TERMINATED BY "column_separator"] [FORMAT AS "file_type"] [BASE_PATH AS "base_path"] [(column_list)] [SET (k1 = func(k2))] ``` - **base_path**: the base path property for partition discovery like in Spark We can enable recursively list files under input path by default and list additional partitioned columns in **column_list**. In my opinion, the first way is more explicit but the alternative seems more convenient for users.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
