This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 232d17efea [Enhancement](sparkload) cast the src slot types of bitmap
columns to `bitmap` when FE push tasks in spark load (#12394)
232d17efea is described below
commit 232d17efeab87b7bf1668e03f98dc1d4908a3faf
Author: spaces-x <[email protected]>
AuthorDate: Wed Sep 7 10:07:38 2022 +0800
[Enhancement](sparkload) cast the src slot types of bitmap columns to
`bitmap` when FE push tasks in spark load (#12394)
In the current spark load implementation, the types of source data, that BE
reads from the Broker, are all set to varchar.
However, the two types of varchar and bitmap are not compatible anymore
after version 1.1.0, which will cause spark load failure.
An example of spark load error message:
detailMessage = type not match, originType=VARCHAR(*), targeType=BITMAP
Describe your changes.
Set the src type of the bitmap columns from varchar to bitmapwhen fe
pushtasks.
---
.../main/java/org/apache/doris/load/loadv2/SparkLoadJob.java | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
index 22cde5328e..2d1d2adef9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
@@ -903,10 +903,18 @@ public class SparkLoadJob extends BulkLoadJob {
Map<String, SlotDescriptor> srcSlotDescByName = Maps.newHashMap();
for (Column column : columns) {
SlotDescriptor srcSlotDesc =
descTable.addSlotDescriptor(srcTupleDesc);
-
srcSlotDesc.setType(ScalarType.createType(PrimitiveType.VARCHAR));
srcSlotDesc.setIsMaterialized(true);
srcSlotDesc.setIsNullable(true);
- srcSlotDesc.setColumn(new Column(column.getName(),
PrimitiveType.VARCHAR));
+
+ if (column.getDataType() == PrimitiveType.BITMAP) {
+ // cast to bitmap when the target column type is bitmap
+
srcSlotDesc.setType(ScalarType.createType(PrimitiveType.BITMAP));
+ srcSlotDesc.setColumn(new Column(column.getName(),
PrimitiveType.BITMAP));
+ } else {
+
srcSlotDesc.setType(ScalarType.createType(PrimitiveType.VARCHAR));
+ srcSlotDesc.setColumn(new Column(column.getName(),
PrimitiveType.VARCHAR));
+ }
+
params.addToSrcSlotIds(srcSlotDesc.getId().asInt());
srcSlotDescByName.put(column.getName(), srcSlotDesc);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
