kz930 commented on code in PR #8341:
URL: https://github.com/apache/texera/pull/8341#discussion_r4214570854
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csvOld/CSVOldScanSourceOpDesc.scala:
##########
@@ -77,6 +81,81 @@ class CSVOldScanSourceOpDesc extends ScanSourceOpDesc {
)
}
+ override def standaloneSourcePath(): Option[String] = fileName
+
+ override def standaloneHelpers(): Seq[String] =
Seq(StandaloneHelpers.AttributeCasts)
+
+ override def generateStandaloneCode(): String = {
+ // First character, empty means comma — the same resolution the reader
below does —
+ // and escaped, so every value the field accepts survives being spliced
into Python.
+ // See CSVScanSourceOpDesc for what handing pandas the raw value did.
+ val sep =
customDelimiter.filter(_.nonEmpty).getOrElse(",").charAt(0).toString
+ val encoding = fileEncoding.toString.replace("_", "-").toLowerCase
+ val headerArg = if (hasHeader) "0" else "None"
+
+ val args = scala.collection.mutable.ArrayBuffer[String]()
+ args += s"filepath_or_buffer=$SourceFilePlaceholder"
+ args += s"sep=${pyStringLiteral(sep)}"
+ args += s"""encoding=${pyStringLiteral(encoding)}"""
+ args += s"header=$headerArg"
+
+ // This reader has NO missing value at all: scala-csv hands an omitted
field
+ // back as the empty string, and every other text stands for itself, so a
+ // blank cell is "" and even widens its column to STRING. pandas instead
+ // reads a list of words as missing by default, "NA" and "null" among them,
+ // and reads a blank as NaN. Dropping the default list settles both, and no
+ // na_values is named — where the other CSV readers null a blank, this one
+ // keeps it. See CSVScanSourceOpDesc.
+ args += "keep_default_na=False"
+
+ // Read a LONG column as the nullable integer, so a hole does not widen it
+ // through a float and round the values it carries, and a timestamp as its
+ // text, parsed below the way the engine parses it. By position, header or
+ // not, since a blank header is not yet the schema's name. See
+ // CSVScanSourceOpDesc.
+ val readAs = Map(AttributeType.LONG -> "Int64", AttributeType.TIMESTAMP ->
"object")
+ val typedColumns: Seq[String] =
+ Try(sourceSchema()).toOption.toSeq.flatMap(
+ _.getAttributes.zipWithIndex.flatMap {
+ case (attribute, i) => readAs.get(attribute.getType).map(d =>
s"""$i: "$d"""")
+ }
+ )
+ if (typedColumns.nonEmpty) args += s"dtype={${typedColumns.mkString(",
")}}"
+
+ // Clamped, as in the newer CSV scan: pandas rejects a negative `nrows`
where
+ // the executor's `take` keeps no rows, and only the editor refuses one.
+ offset.map(_.max(0)).foreach { o =>
+ // Counted in Long past the header and asked of each line, as in the
newer
+ // CSV scan.
+ if (hasHeader) args += s"skiprows=lambda _i: 0 < _i <= ${o.toLong}"
+ else args += s"skiprows=$o"
+ }
+ limit.map(_.max(0)).foreach(l => args += s"nrows=$l")
+
+ val readCall = s"out1df = pd.read_csv(${args.mkString(", ")})"
Review Comment:
Fixed in 280aaebd1. Without a header, both readers now pass `names=` by
position, so an offset past EOF reads zero rows with the schema's columns. Each
spec has a parity test against the engine, and both tests fail without the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
