olaky commented on code in PR #40035:
URL: https://github.com/apache/spark/pull/40035#discussion_r1112705403
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -230,6 +230,14 @@ object FileSourceStrategy extends Strategy with
PredicateHelper with Logging {
case MetadataStructColumn(attr) => attr
}
+ // We divide metadata columns into two categories: constant and
generated.
+ // For constant metadata columns, we create these attributes as
non-nullable
+ // when passing to readers, since the values remain persistent in the
files.
+ // For generated metadata columns, they are set as nullable when passed
to readers,
+ // as their values are generated on the fly when reading and could be
null.
+ // After reading, values will be replaced.
+ // So all metadata columns will be non-null in the returned output.
Review Comment:
```suggestion
// All metadata columns will be non-null in the returned output.
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -230,6 +230,14 @@ object FileSourceStrategy extends Strategy with
PredicateHelper with Logging {
case MetadataStructColumn(attr) => attr
}
+ // We divide metadata columns into two categories: constant and
generated.
+ // For constant metadata columns, we create these attributes as
non-nullable
+ // when passing to readers, since the values remain persistent in the
files.
Review Comment:
```suggestion
// when passing to readers, since the values are always available.
```
The values are actually not always stored, for example the path of a file is
not stored in the file
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:
##########
@@ -230,6 +230,14 @@ object FileSourceStrategy extends Strategy with
PredicateHelper with Logging {
case MetadataStructColumn(attr) => attr
}
+ // We divide metadata columns into two categories: constant and
generated.
+ // For constant metadata columns, we create these attributes as
non-nullable
+ // when passing to readers, since the values remain persistent in the
files.
+ // For generated metadata columns, they are set as nullable when passed
to readers,
+ // as their values are generated on the fly when reading and could be
null.
Review Comment:
```suggestion
// as the values will be null when trying to read the missing column
from the file. They are then replaced by the actual values later in the process.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
