This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2331ce1 [Bug]Parquet map/list/struct structure recognize (#4968)
2331ce1 is described below
commit 2331ce10f18074d2a135ad8079c6df863719d920
Author: xinghuayu007 <[email protected]>
AuthorDate: Sat Nov 28 09:56:29 2020 +0800
[Bug]Parquet map/list/struct structure recognize (#4968)
When a parquet file contains a `Map/List/Struct` structure, Doris can not
recognize the column correctly,
and throws exception 'Invalid column: xxxx', that means Doris can not find
the column.
The `Map` structure will be recognized into two columns: `key and value`.
The follow is the schema of a parquet file recognized by Doris. This patch
tries to solve this problem.
---
be/src/exec/parquet_reader.cpp | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/be/src/exec/parquet_reader.cpp b/be/src/exec/parquet_reader.cpp
index 6ed2020..0b38c16 100644
--- a/be/src/exec/parquet_reader.cpp
+++ b/be/src/exec/parquet_reader.cpp
@@ -68,7 +68,11 @@ Status ParquetReaderWrap::init_parquet_reader(const
std::vector<SlotDescriptor*>
auto *schemaDescriptor = _file_metadata->schema();
for (int i = 0; i < _file_metadata->num_columns(); ++i) {
// Get the Column Reader for the boolean column
- _map_column.emplace(schemaDescriptor->Column(i)->name(), i);
+ if (schemaDescriptor->Column(i)->max_definition_level() > 1) {
+
_map_column.emplace(schemaDescriptor->Column(i)->path()->ToDotVector()[0], i);
+ } else {
+ _map_column.emplace(schemaDescriptor->Column(i)->name(), i);
+ }
}
_timezone = timezone;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
