liujiayi771 commented on code in PR #4779:
URL: https://github.com/apache/incubator-gluten/pull/4779#discussion_r1521626189
##########
cpp/velox/compute/WholeStageResultIterator.cc:
##########
@@ -152,8 +152,28 @@ WholeStageResultIterator::WholeStageResultIterator(
auto partitionColumn = partitionColumns[idx];
std::unordered_map<std::string, std::optional<std::string>>
partitionKeys;
constructPartitionColumns(partitionKeys, partitionColumn);
- auto split =
std::make_shared<velox::connector::hive::HiveConnectorSplit>(
- kHiveConnectorId, paths[idx], format, starts[idx], lengths[idx],
partitionKeys);
+ std::shared_ptr<velox::connector::ConnectorSplit> split;
+ if (auto icebergSplitInfo =
std::dynamic_pointer_cast<IcebergSplitInfo>(scanInfo)) {
+ // Set Iceberg split.
+ std::unordered_map<std::string, std::string>
customSplitInfo{{"table_format", "hive-iceberg"}};
+ auto deleteFilesFind =
icebergSplitInfo->deleteFilesMap.find(paths[idx]);
+ auto deleteFiles = deleteFilesFind !=
icebergSplitInfo->deleteFilesMap.end() ? deleteFilesFind->second
Review Comment:
@yma11 This is also a feasible solution, but not all data files have a
delete file, and some data files may have multiple delete files. We would need
a two-dimensional vector to maintain this relationship, and the two-dimensional
vector would need to have empty vector for data files that do not have a
corresponding delete file. Do you think this is a better solution? I'm OK to
change to using this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
