malinjawi commented on code in PR #12836:
URL: https://github.com/apache/gluten/pull/12836#discussion_r3869721487
##########
gluten-delta/src-delta33/main/scala/org/apache/gluten/delta/DeltaDeletionVectorScanInfo.scala:
##########
@@ -47,7 +49,11 @@ object DeltaDeletionVectorScanInfo {
hasDeletionVector: Boolean,
rowIndexFilterType: RowIndexFilterType,
cardinality: Long,
- serializedDeletionVector: Array[Byte])
+ deletionVectorPayload: DeletionVectorPayload) {
+ def serializedDeletionVector: Array[Byte] =
deletionVectorPayload.materialize()
+
+ def isPayloadMaterialized: Boolean = deletionVectorPayload.isMaterialized()
Review Comment:
Removed `isPayloadMaterialized` from both the Delta 3.3 and 4.0
implementations. Thanks.
##########
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/DeltaLocalFilesNode.java:
##########
@@ -79,24 +79,76 @@ public enum RowIndexFilterType {
IF_NOT_CONTAINED
}
+ /**
+ * Serializable source for a deletion-vector payload.
+ *
+ * <p>The source travels inside a Spark input partition. Implementations may
therefore defer
+ * remote I/O until {@link #materialize()} is called while the split is
converted to protobuf on
+ * an executor. The returned byte array must not be modified: protobuf wraps
it without copying.
+ */
+ public interface DeletionVectorPayload extends Serializable {
+ byte[] materialize();
+
+ /** Returns whether the payload bytes are already resident in this object.
*/
+ boolean isMaterialized();
+ }
+
+ /** A payload source for inline DVs whose bytes are already present in Delta
metadata. */
+ public static final class SerializedDeletionVectorPayload implements
DeletionVectorPayload {
Review Comment:
Agreed. Renamed it to `InMemoryDeletionVectorPayload` and updated all
references and tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
