This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7fedfdcf6a [fix](spark load)The where condition does not take effect
when spark load loads the file (#13803)
7fedfdcf6a is described below
commit 7fedfdcf6a3cbe96b6d71fff460f27c96a9dbcae
Author: jiafeng.zhang <[email protected]>
AuthorDate: Tue Nov 1 23:01:45 2022 +0800
[fix](spark load)The where condition does not take effect when spark load
loads the file (#13803)
---
.../src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
index d64c1080c8..3c6a27157e 100644
--- a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
+++ b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
@@ -724,6 +724,9 @@ public final class SparkDpp implements java.io.Serializable
{
);
Dataset<Row> dataframe = spark.createDataFrame(rowRDD, srcSchema);
+ if (!Strings.isNullOrEmpty(fileGroup.where)) {
+ dataframe = dataframe.where(fileGroup.where);
+ }
return dataframe;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
