Hi Roman, Igal (@ below) Thank you for your answer. I don't think I'll have access to flink's lib folder given it's a shared Cloudera cluster. The only thing I could think of is to not include com.google.protobuf in the classloader.parent-first-patterns.additional setting, and including protobuf-java 3.7.1 in the uber jar.
I created a jira for this just now + a discuss thread on the dev group https://issues.apache.org/jira/browse/FLINK-26537 Hi @Igal Shilman <igal.shil...@gmail.com> , is the plugin solution outlined by Roman something that fits in better with Statefun than having the creators of uber .jars be responsible for using a statefun-compatible protobuf-java? Kind regards Fil On Tue, 8 Mar 2022 at 14:02, Roman Khachatryan <ro...@apache.org> wrote: > Hi Filip, > > Have you tried putting protobuf-java 3.7.1 into the Flink's lib/ folder? > Or maybe re-writing the dependencies you mentioned to be loaded as > plugins? [1] > > I don't see any other ways to solve this problem. > Probably Chesnay or Seth will suggest a better solution. > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/ > > > Regards, > Roman > > On Fri, Mar 4, 2022 at 9:54 AM Filip Karnicki <filip.karni...@gmail.com> > wrote: > > > > Hi All! > > > > We're running a statefun uber jar on a shared cloudera flink cluster, > the latter of which launches with some ancient protobuf dependencies > because of reasons[1]. > > > > Setting the following flink-config settings on the entire cluster > > > > classloader.parent-first-patterns.additional: > org.apache.flink.statefun;org.apache.kafka;com.google.protobuf > > > > causes these old protobuf dependencies to get loaded over statefun's > protobuf-java 3.7.1, and NoSuchMethod exceptions occur. > > > > We hacked together a version of statefun that doesn't perform the check > whether the classloader settings contain the three patterns from above, and > as long as our job uses protobouf-java 3.7.1 and the com.google.protobuf > pattern is not present in the classloader.parent-first-patterns.additional > setting, then all is well. > > > > Aside from removing old hadoop from the classpath, which may not be > possible given that it's a shared cluster, is there anything we can do > other than adding a configurable override not to perform the config check > in StatefulFunctionsConfigValidator to an upcoming statefun core release? > > > > Many thanks > > Fil > > > > > > [1] We're still trying to find out if it's absolutely necessary to have > these on the classpath. >
