kz930 commented on code in PR #8704:
URL: https://github.com/apache/texera/pull/8704#discussion_r4252377452
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/json/JSONLScanSourceOpDesc.scala:
##########
@@ -87,8 +265,12 @@ class JSONLScanSourceOpDesc extends ScanSourceOpDesc {
val allFields: ArrayBuffer[Map[String, String]] = ArrayBuffer()
val startOffset = windowOffset
+ // A window of no rows is still a window on this file, and the file's
columns
+ // do not depend on how many of its rows were asked for. Reading the sample
+ // through the limit left a Limit of 0 nothing to infer from, and the
operator
+ // declared a schema of no columns at all.
val endOffset =
- startOffset + inferSampleSize
+ startOffset + windowLimit.filter(_ >
0).getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
Review Comment:
Fixed in d50d41016. When the window sample is empty, the schema now samples
from the top of the file, and the offset=10 repro is a test in
JSONLScanSourceOpDescSpec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
