comphead commented on code in PR #5196: URL: https://github.com/apache/datafusion-comet/pull/5196#discussion_r3696090279
########## docs/source/faq.md: ########## @@ -0,0 +1,231 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Frequently Asked Questions + +Short answers to the questions we see most often from Comet users, each linking to the guide that +covers the topic in full. As with the rest of the unversioned pages on this site, links point into +the development snapshot of the user guide. If you are running a released version, the equivalent +pages for it are linked from the [User Guide](user-guide/index.md) index. + +## Getting Started + +### Do I need to change my queries or my code to use Comet? + +No. Comet is a Spark plugin. You add the Comet jar to an existing Spark job, set +`spark.plugins=org.apache.spark.CometPlugin`, and Comet rewrites the physical plan behind the scenes. +Your DataFrame and SQL code is unchanged. See the [installation guide](user-guide/latest/installation.md) +for the full set of properties, including the off-heap memory and shuffle manager settings that most +deployments want. + +### How do I confirm that Comet is actually accelerating my query? + +Three checks, from cheapest to most detailed: + +1. Look for `INFO core/src/lib.rs: Comet native library version <version> initialized` in the + executor logs. If it is absent, the native library never loaded and nothing is being accelerated. +2. Read `spark.conf.get("spark.comet.version")` to confirm which build is loaded. +3. Run `EXPLAIN FORMATTED` on the query and look for `Comet*` operators in the plan. Anything without + the `Comet` prefix is running on unmodified Spark. Review Comment: would be nice having an example that shows a plan with fallbacks ``` val plan = df.queryExecution.executedPlan val info = new org.apache.comet.ExtendedExplainInfo() println(info.generateExtendedInfo(plan)) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
