MaxGekk commented on code in PR #36435:
URL: https://github.com/apache/spark/pull/36435#discussion_r865208393
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
Review Comment:
Looking configs around, this should be shifted by 2 spaces left?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala:
##########
@@ -2693,6 +2693,17 @@ abstract class CSVSuite
assert(df.schema == expected)
checkAnswer(df, Row(1, null) :: Nil)
}
+
+ withSQLConf(
+ SQLConf.LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION.key ->
true.toString) {
Review Comment:
nit:
```suggestion
withSQLConf(SQLConf.LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION.key
-> "true") {
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
+ .internal()
+ .doc(
+ "When true, the nullability in the user-specified schema for " +
+ "`DataFrameReader.schema(schema).json(jsonDataset)` and " +
+ "`DataFrameReader.schema(schema).csv(csvDataset)` is respected.
Otherwise, they are " +
+ "turned to a nullable schema forcely.")
Review Comment:
unindent by 2 spaces
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
+ .internal()
+ .doc(
+ "When true, the nullability in the user-specified schema for " +
+ "`DataFrameReader.schema(schema).json(jsonDataset)` and " +
+ "`DataFrameReader.schema(schema).csv(csvDataset)` is respected.
Otherwise, they are " +
+ "turned to a nullable schema forcely.")
Review Comment:
forcely -> forcibly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
