yuqi1129 opened a new issue, #8916:
URL: https://github.com/apache/gravitino/issues/8916
### Describe the subtask
When we try to create a table by `lance-spark` like
```python
from pyspark.sql import SparkSession
import os
import logging
logging.basicConfig(level=logging.INFO)
os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars
/Users/yuqi/Downloads/lance-spark-bundle-3.5_2.12-0.0.10.jar,/Users/yuqi/Downloads/lance-namespace-hive2-0.0.14.jar
--master local[1] pyspark-shell"
spark = SparkSession.builder \
.appName("gcs_fielset_test") \
.config("spark.sql.catalog.lance",
"com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \
.config("spark.sql.catalog.lance.impl", "rest") \
.config("spark.executor.memory", "3g") \
.config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \
.config("spark.sql.catalog.lance.headers.x-lance-root-catalog",
"lance_catalog") \
.config("spark.sql.catalog.lance.headers.x-lance-table-location",
"/tmp/lance_catalog/schema/users.lance") \
.config("spark.sql.defaultCatalog", "lance") \
.getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")
results = spark.sql("""
CREATE TABLE schema.users (
id BIGINT NOT NULL,
name STRING,
email STRING
);
""")
results.show()
```
It will output the path `/tmp/lance_catalog/schema/users.lance` already
exists. It seems that Spark will try to create the lance directory althrough we
have created it by Gravitiono.
### Parent issue
#8828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
