carloea2 commented on code in PR #8704:
URL: https://github.com/apache/texera/pull/8704#discussion_r4252179358
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpDesc.scala:
##########
@@ -87,8 +265,12 @@ class JSONLScanSourceOpDesc extends ScanSourceOpDesc {
val allFields: ArrayBuffer[Map[String, String]] = ArrayBuffer()
val startOffset = windowOffset
+ // A window of no rows is still a window on this file, and the file's
columns
+ // do not depend on how many of its rows were asked for. Reading the sample
+ // through the limit left a Limit of 0 nothing to infer from, and the
operator
+ // declared a schema of no columns at all.
val endOffset =
- startOffset + inferSampleSize
+ startOffset + windowLimit.filter(_ >
0).getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
Review Comment:
The zero-row schema fix still misses a valid offset beyond EOF. On a two-row
JSONL file with id and name, offset=10 returns a schema with no columns; the
limit=0 control keeps both columns. Reproduced in JSONLScanSourceOpDescSpec.
Please infer from a nonempty file sample when this window is empty, as the CSV
fix does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
