goutamadwant opened a new issue, #12334:
URL: https://github.com/apache/seatunnel/issues/12334
### Search before asking
- [x] I searched the existing issues and pull requests for this failure.
CopyTransform vector support (#11222) and the broader row-immutability
discussion (#10942) do not address repeated reads inside one SQL expression
pipeline.
### What happened
SQL vector functions consume the position of the input `ByteBuffer` while
converting it to a float array. Reading the same vector again can throw
`BufferUnderflowException`. This affects multiple projections on one vector
field, distance functions given the same buffer as both operands, and
downstream reuse of a vector passed through a SQL transform.
The problem is in `VectorFunction.convertToFloatArray`: it passes the
caller's buffer directly to the consuming `VectorUtils.toFloatArray` utility.
It is not necessary to change the utility's contract or deep-copy the row to
fix this call site.
### SeaTunnel Version
Reproduced against `dev` commit `1325a44b2f4013198fdb2c12f9df345de1cde4bc`
(3.0.0-SNAPSHOT). Released versions were not separately tested.
### Reproduction
The following calls use the existing production functions:
```java
ByteBuffer vector = VectorUtils.toByteBuffer(new Float[] {3.0f, 4.0f});
List<Object> args = Collections.<Object>singletonList(vector);
VectorFunction.vectorDims(args); // returns 2, but consumes the buffer
VectorFunction.vectorDims(args); // throws BufferUnderflowException
```
Expected: both calls return 2 and leave the input position, limit, mark,
byte order and contents unchanged. A failed read should also leave the original
buffer state unchanged.
This was also reproduced through `SQLTransform`, not only direct helper
calls. In the existing vector E2E fixture, a first SQL transform normalizes
`array<float>` inputs into vector buffers. A second evaluates:
```sql
SELECT VECTOR_DIMS(vector_field) AS original_dim,
VECTOR_DIMS(vector_field) AS repeated_dim,
L1_DISTANCE(vector_field, vector_field) AS self_distance
FROM dual
```
### SeaTunnel Config
The complete regression configuration, including FakeSource rows, both SQL
transforms and Assert rules, is
[func_vector.conf](https://github.com/goutamadwant/seatunnel/blob/03e7a6facadca546490af25099aaabd3b51c4995/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/sql_transform/func_vector.conf).
### Running Command
With FakeSource, Assert and the transform plugin installed, save that
configuration as `vector-read.conf` and run:
```sh
sh bin/seatunnel.sh -c vector-read.conf -e local
```
Regression tests on the fix branch:
```sh
./mvnw -B -pl seatunnel-transforms-v2 -am package \
'-Dtest=VectorFunctionTest,SQLVectorFunctionTest' \
-Dsurefire.failIfNoSpecifiedTests=false -Dskip.ui=true
```
### Error Exception
The SQL path reports `TRANSFORM_COMMON-06`, caused by
`java.nio.BufferUnderflowException` while decoding the repeated vector read.
Before changing production code, the extended regression suite failed on
both Java 8 and Java 11: 32 tests, 1 failure and 12 errors on each JDK. The 19
existing cases passed; all 13 added invocations exposed the mutation or
underflow. A separate failed-read check observed the original position advance
from 4 to 12 before throwing.
### Proposed fix, before and after
Read an ordered duplicate at the SQL conversion boundary:
`buffer.duplicate().order(buffer.order())`.
- Before: a vector read consumes shared caller state and can break another
expression or downstream consumer.
- After: each read has its own cursor, so repeated expressions,
self-distance operands and pass-through vectors can reuse the original input.
- Preserve `VectorUtils` consuming behavior for other callers, existing
buffer dimensions and malformed-buffer exceptions. Do not rewind or reinterpret
input buffers.
- Add coverage for heap, direct, read-only, sliced and offset buffers, both
byte orders, failed reads, zero normalization and no-op reductions. Extend the
existing SQL E2E fixture and EN/ZH docs.
This adds a buffer view, not a full byte-buffer copy. It does not provide
protection against concurrent writes to shared underlying bytes.
### Validation and advantages
Full transform-reactor builds pass on Java 8 and Java 11, with 1,170
transform tests on each and no failures, errors or skips. A combined
integration check with the separate vector-arithmetic fix also passes on both
JDKs.
An isolated Java 11 Zeta runtime check reproduced the buffer failure with
the original transform JAR and passed with the fixed combined JAR. That check
used an earlier 3.0.0-SNAPSHOT distribution with the current transform JAR
overlaid; it is not the complete Flink/Spark/Testcontainers matrix.
The benefit is reliable reuse of existing vector data within SQL processing,
without adding a row-copy requirement or changing connector APIs.
### Breaking changes
No API, configuration, schema, dependency or state-format change. This
corrects a read-side effect; code intentionally depending on SQL vector
functions advancing an input cursor will no longer observe that side effect.
Vector arithmetic is outside this issue.
### Zeta or Flink or Spark Version
Zeta 3.0.0-SNAPSHOT for the targeted runtime check. Flink/Spark integration
execution is not claimed.
### Java or Scala Version
Java 8u172 and Temurin 11.0.19 for reproduction and full transform tests.
Java 11 for the isolated Zeta runtime check.
### Are you willing to submit PR?
- [x] Yes, the fix and regression coverage are prepared.
### Code of Conduct
- [x] I agree to follow the project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
