jinwenjie123 commented on issue #1552:
URL:
https://github.com/apache/datafusion-comet/issues/1552#issuecomment-2759448204
> > `org.apache.comet.serde.QueryPlanSerde Comet native execution is
disabled due to: unsupported Spark partitioning:
org.apache.spark.sql.catalyst.plans.physical.RangePartitioning`
> > `org.apache.comet.serde.QueryPlanSerde Comet native execution is
disabled due to: unsupported Spark data type:
MapType(StringType,StringType,true)`
> > Is it because I missed to add some required libraries ?
>
> No, it's not because of missing libraries. The messages say you're
encountering something in the Spark plan that Comet does not support
(RangePartitioning and the MapType specifically). This is odd because TPC-H
does not have `MapType`s. Do you have a modified version of the queries and
data?
Hi @parthchandra ,
I checked the process, and it seems that the unsupported type is introduced
by the Delta conversion process added by our benchmark. After disabling Delta,
the message no longer appeared.
However, when testing TPC-H Q1 in the AWS environment, I only observed a 10%
improvement, whereas the documentation claims that Comet can achieve a 549%
improvement. While I am running it on AWS with Spark 3.4.3, and the
documentation references Spark 3.5.4 running locally, I’m not sure why I’m not
seeing a similar level of improvement.
When I built the Comet jar, I changed the release target of the Makefile as
the following and run command `make release PROFILES="-Pspark-3.4
-Pscala-2.13"` to build the jar
`release:
cd native && cargo zigbuild --target x86_64-unknown-linux-gnu.2.26
--release $(FEATURES_ARG)
mkdir -p common/target/classes/org/apache/comet/linux/amd64
cp native/target/x86_64-unknown-linux-gnu/release/libcomet.so
common/target/classes/org/apache/comet/linux/amd64
jar -cf common/target/comet-native-x86_64.jar \
-C common/target/classes/org/apache/comet linux
./mvnw install -Prelease -DskipTests $(PROFILES)`
Dose the above changes could cause some issue ?
Thanks !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
