gustavoatt commented on code in PR #7352:
URL: https://github.com/apache/iceberg/pull/7352#discussion_r1256437201
##########
core/src/main/java/org/apache/iceberg/StaticDataTask.java:
##########
@@ -71,8 +71,10 @@ public List<DeleteFile> deletes() {
@Override
public CloseableIterable<StructLike> rows() {
- StructProjection projection = StructProjection.create(tableSchema,
projectedSchema);
- Iterable<StructLike> projectedRows =
Iterables.transform(Arrays.asList(rows), projection::wrap);
+ Iterable<StructLike> projectedRows =
Review Comment:
@szehon-ho I believe it work in Spark, because the Spark API [iterates one
row at a
time](https://github.com/apache/iceberg/blob/master/spark/v3.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseDataReader.java#L103),
using [this
iterator](https://github.com/apache/iceberg/blob/master/spark/v3.1/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java#L87).
I tested changing `reuseContainer` in `ManifestReader`, but it does not
work. I believe the problem is with always using the same `StructProjection`.
One alternative I see is modifying the `DataTask` interface and add a
`reuseContainer` method there that that Generic can then set to false but it
will be set to true everywhere else. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
