qingfei1994 commented on issue #5079:
URL: https://github.com/apache/paimon/issues/5079#issuecomment-2681885550
> I agree that `user-defined aggregate-functions` should be validated in
advance, rather than during the actual write stage. Perhaps we could add this
validation when creating the `Source / Sink`, so that it can be detected during
the job submission phase.
After investigation, I think the reason it's validated in the write stage is
that the MergeFunctionFactory has been lazily created during the write stage.
KeyValueFileStoreWrite.java
` return new MergeTreeWriter(
bufferSpillable(),
options.writeBufferSpillDiskSize(),
options.localSortMaxNumFileHandles(),
options.spillCompressOptions(),
ioManager,
compactManager,
restoredMaxSeqNumber,
keyComparator,
mfFactory.create(),
writerFactory,
options.commitForceCompact(),
options.changelogProducer(),
restoreIncrement,
UserDefinedSeqComparator.create(valueType, options));`
Do you think it's appropriate to add a validate function in
MergeFunctionFactory something like this, to validate the creation of
aggregators in the constructors of KeyValueFileStoreWrite? please suggest.
`
public interface MergeFunctionFactory<T> extends Serializable {
....
default void validate() {
// TODO
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
