David Mollitor created SPARK-59615:
--------------------------------------
Summary: Avoid an intermediate buffer in
VectorizedPlainValuesReader.readGeoData
Key: SPARK-59615
URL: https://issues.apache.org/jira/browse/SPARK-59615
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. Problem
{{VectorizedPlainValuesReader.readGeoData}} (the vectorized PLAIN-encoding read
path for GEOMETRY/GEOGRAPHY columns) buffers every converted value for the
whole page into a {{ByteBufferOutputStream}} created with no initial capacity,
then copies it out into the column vector:
{code:java}
int base = v.arrayData().getElementsAppended();
int dataLen = 0;
final int intSize = 4;
ByteBuffer lenBuffer = ByteBuffer.allocate(intSize);
ByteBufferOutputStream out = new ByteBufferOutputStream();
for (int i = 0; i < total; i++) {
int len = readInteger();
byte[] physicalValue = converter.convert(in.readNBytes(len), srid);
v.putArray(rowId + i, base + dataLen + intSize, physicalValue.length);
lenBuffer.putInt(0, physicalValue.length);
out.write(lenBuffer.array());
out.write(physicalValue);
dataLen += intSize + physicalValue.length;
}
out.close();
v.arrayData().appendBytes(dataLen, out.toByteArray(), 0);
{code}
This has two costs:
* The no-arg {{ByteBufferOutputStream}} starts at the JDK default of 32 bytes
and reallocates and copies its backing array on the order of {{log2(pageBytes /
32)}} times as it accumulates the page; the data is then copied twice more
({{toByteArray()}} and the final {{appendBytes}}).
* A redundant 4-byte length prefix is written before each value. The element
length is already recorded by {{putArray}}'s third argument, and {{putArray}}
points _past_ the prefix, so those bytes are never read.
The sibling reader {{VectorizedDeltaByteArrayReader.readGeoData}} already
avoids both: it appends each converted value directly to the column vector's
{{arrayData()}} with no intermediate buffer and no length prefix. The two
implementations have diverged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
