Igosuki opened a new issue #1203:
URL: https://github.com/apache/arrow-datafusion/issues/1203
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
Haven't see that anywhere in the code, so please let me know if I overlooekd
it.
Spark supports pseudo columns, a pseudo column is a column that has a single
value for an entire partition, and can be used to filter data without reading
any physical files.
**Describe the solution you'd like**
reading `basepath` which contains two directories `foo=a` and `foo=b` gives
me a dataframe with the utf8 column `foo` merged into the schema, and a
partition for each value.
If I filter on that column, for instance `select * from basepathtable where
foo = "a"` only scans the files in that directory.
**Describe alternatives you've considered**
Currently AFAIK, datafusion requires you to know the full list of partition
directories and read from each individually.
**Additional context**
Example pyspark code where basepath contains directories like `dt={*}`
```python
df = self.spark.read.option("basePath",
basepath).format("parquet").load(path)
filtered = df.filter((col('dt') >= start) & (col('dt') <= end))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
