This is an automated email from the ASF dual-hosted git repository.
chitralverma pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/griffin.git
The following commit(s) were added to refs/heads/master by this push:
new bc40c97 [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x
environment
bc40c97 is described below
commit bc40c97974fd98272c75b731a4a588b1746af111
Author: Lipeng Zhu <[email protected]>
AuthorDate: Mon Jan 24 10:36:22 2022 +0530
[GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
**What changes were proposed in this pull request?**
Built in Avro format is released in Spark
2.4.0,https://issues.apache.org/jira/browse/SPARK-24768
For Griffin, we still need to convert the Avro to com.databricks.spark.avro
in Spark 2.3.x environment.
**Does this PR introduce any user-facing change?**
No.
**How was this patch tested?**
Unit Tests
Closes #598 from lipzhu/GRIFFIN-369.
Lead-authored-by: Lipeng Zhu <[email protected]>
Co-authored-by: Chitral Verma <[email protected]>
Co-authored-by: lipzhu <[email protected]>
Signed-off-by: chitralverma <[email protected]>
---
.../measure/datasource/connector/batch/FileBasedDataConnector.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
b/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
index 2e4f482..4f75812 100644
---
a/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
+++
b/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
@@ -79,7 +79,8 @@ case class FileBasedDataConnector(
SupportedFormats.contains(format),
s"Invalid format '$format' specified. Must be one of
${SupportedFormats.mkString("['", "', '", "']")}")
- if (format.equalsIgnoreCase("avro") && sparkSession.version < "2.3.0") {
+ // Use old implementation for AVRO format if current spark version is not
2.4.x and above
+ if ("avro".equalsIgnoreCase(format) && sparkSession.version < "2.4.0") {
format = "com.databricks.spark.avro"
}
