kbendick edited a comment on pull request #3857: URL: https://github.com/apache/iceberg/pull/3857#issuecomment-1007663657
> > Is this due to Flink 1.13 only (e.g. fixed / not known to occur in 1.14 or 1.12) or did you just choose to implement in 1.13 for another reason? > > The fix for `SerializableTable` should be okay for all the flink versions. I'm trying to add this test casest flink 1.13 firstly, it should also be applied to other flink versions. I think I will port those to flink 1.12 & flink 1.13 then. Perfect. I just wasn't sure as sometimes people do 1.14 for new PRs (or Spark 3.2 for example). Since it wasn't latest, that's what made me ask if maybe this was 1.13 specific. But it's absolutely ok to do it in 1.13 first for review purposes. Thanks for clarifying! > > The SerializableSupplier is written as is to avoid calls to get on the user-side. > > I think the `SerializableSupplier` is mainly designed for resolving the hadoop `Configuration` serialization & deserialization issues. We maintain all the key-value pairs inside the in-memory HashMap from the hadoop `Configuration` and then restore to the hadoop `Configuration` once we've transferred those bytes to engines's DAG operators ( Saying `SerializableSupplier#get`). > > The current patch don't change the previous intentional design, instead it is trying to implement it in another way to fix the flink kyro serialization issues. That's correct. My mistake. I meant the current usage that has the `::get` call inside of `serializeConfWith`. A minor historical detail. You are right, that the SerializableSupplier is to deal with SerDe issues. This doesn't change that. Sorry for any confusion. 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
