JingsongLi commented on code in PR #9598:
URL: https://github.com/apache/paimon/pull/9598#discussion_r3931980818
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonPartitionReaderFactory.scala:
##########
@@ -29,13 +30,19 @@ import java.util.Objects
case class PaimonPartitionReaderFactory(
readBuilder: ReadBuilder,
metadataColumns: Seq[PaimonMetadataColumn] = Seq.empty,
- blobAsDescriptor: Boolean)
+ blobAsDescriptor: Boolean,
+ uriReaderFactory: UriReaderFactory = null)
Review Comment:
[P1] Propagate the descriptor reader through every Spark read path
This new dependency defaults to null, and
PaimonMicroBatchStream#createReaderFactory still constructs
PaimonPartitionReaderFactory with only readBuilder/blobAsDescriptor. Every
Structured Streaming read therefore takes the uriReaderFactory == null branch
and keeps the BlobRef reader built from the target table FileIO, so descriptors
that require blob-descriptor-source-table still fail on toData(). The
postpone.merge-on-read path has the same gap: SortedBucketMergeIterator
converts currentReader.next() directly and MergePlan carries only
blobAsDescriptor. Please build/pass the source UriReaderFactory through both
paths (and add streaming/postponed-read regressions) instead of silently
falling back to null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
