Hi all, quick follow-up: thank you Philip, you were right ! Indeed, I got rid of this error by putting the converter in the connector's folder.
I also tried something else: create a custom connector and use that custom connector with the custom converter, both loaded as plugins. It also works. Summary: converters are loaded by the connector. If your connector is a plugin, your converter should be as well. If you connector is not a plugin (bundled with your kafka connect distrib), your converter should not be either. Jehan On 17 October 2017 at 14:51, Jehan Bruggeman <jehanbrugge...@gmail.com> wrote: > HI Philip, > > thanks for the reply ! > > This issue looks like it might match my problem indeed. I'll look into it. > > Jehan > > On 16 October 2017 at 20:53, Philip Schmitt <philip.schm...@outlook.com> > wrote: > >> Hi Jehan, >> >> I've run into the same issue last week and also got a "class could not be >> found" error. >> >> Konstantine Karantasis helpfully pointed me towards >> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007 >> >> To test this, I simply copied my SMT jar to the folder of the connector I >> was using and adjusted the plugin.path property. >> >> I haven't fully tested tested it, but in my very quick initial test, it >> seemed to have move past that "class could not be found" issue. >> >> Regards, >> Philip >> ________________________________ >> From: jeh...@gmail.com <jeh...@gmail.com> on behalf of Jehan Bruggeman < >> jehanbrugge...@gmail.com> >> Sent: Monday, October 16, 2017 8:17 AM >> To: users@kafka.apache.org >> Subject: Re: Custom converter with Kafka Connect ? >> >> Hi Randall, >> >> thanks for your reply. I'm not sure about this; what detail could I add >> that would help you figure it out ? >> >> Concerning the classpath: as described in my original email, I'm pretty >> sure the jars are correctly added to the classpath since the classes in >> the >> jar are recognized by Kafka Connect when it starts (they are mentionned in >> the logs, at least). >> >> (It's easier to read here, where I asked the same question: >> https://stackoverflow.com/questions/46712095/using-a-custom- >> converter-with-kafka-connect >> ) >> >> thanks for your help ! >> >> Jehan >> >> On 13 October 2017 at 16:07, Randall Hauch <rha...@gmail.com> wrote: >> >> > On Tue, Oct 10, 2017 at 8:31 AM, Jehan Bruggeman < >> jehanbrugge...@gmail.com >> > > >> > wrote: >> > >> > > Hello, >> > > >> > > I'm trying to use a custom converter with Kafka Connect and I cannot >> seem >> > > to get it right. I'm hoping someone has experience with this and could >> > help >> > > me figure it out ! >> > > >> > > >> > > Initial situation >> > > ================ >> > > >> > > - my custom converter's class path is 'custom.CustomStringConverter'. >> > > >> > > - to avoid any mistakes, my custom converter is currently just a >> > copy/paste >> > > of the pre-existing StringConverter (of course, this will change when >> > I'll >> > > get it to work). >> > > https://github.com/apache/kafka/blob/trunk/connect/api/ >> > > src/main/java/org/apache/kafka/connect/storage/StringConverter.java >> > > >> > > - I have a kafka connect cluster of 3 nodes, The nodes are running >> > > confluent's official docker images ( confluentinc/cp-kafka-connect: >> 3.3.0 >> > > ). >> > > >> > > - Each node is configured to load a jar with my converter in it >> (using a >> > > docker volume). >> > > >> > >> > Can you explain this in more detail? Make sure that you add the JAR to >> the >> > classpath. >> > >> > >> > > >> > > >> > > >> > > What happens ? >> > > ================ >> > > >> > > When the connectors start, they correctly load the jars and find the >> > custom >> > > converter. Indeed, this is what I see in the logs : >> > > >> > > [2017-10-10 13:06:46,274] INFO Registered loader: >> > > PluginClassLoader{pluginLocation=file:/opt/custom-connectors/custom- >> > > converter-1.0-SNAPSHOT.jar} >> > > (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >> oader:199) >> > > [2017-10-10 13:06:46,274] INFO Added plugin >> > 'custom.CustomStringConverter' >> > > (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >> oader:132) >> > > [...] >> > > [2017-10-10 13:07:43,454] INFO Added aliases 'CustomStringConverter' >> and >> > > 'CustomString' to plugin 'custom.CustomStringConverter' >> > > (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >> oader:293) >> > > >> > > I then POST a JSON config to one of the connector nodes to create my >> > > connector : >> > > >> > > { >> > > "name": "hdfsSinkCustom", >> > > "config": { >> > > "topics": "yellow", >> > > "tasks.max": "1", >> > > "key.converter": "org.apache.kafka.connect.stor >> age.StringConverter", >> > > "value.converter": "custom.CustomStringConverter", >> > > "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", >> > > "hdfs.url": "hdfs://hdfs-namenode:8020/hdfs-sink", >> > > "topics.dir": "yellow_storage", >> > > "flush.size": "1", >> > > "rotate.interval.ms": "1000" >> > > } >> > > } >> > > >> > > And receive the following reply : >> > > >> > > { >> > > "error_code": 400, >> > > "message": "Connector configuration is invalid and contains the >> > > following 1 error(s):\nInvalid value custom.CustomStringConverter for >> > > configuration value.converter: Class custom.CustomStringConverter >> could >> > not >> > > be found.\nYou can also find the above list of errors at the endpoint >> > > `/{connectorType}/config/validate`" >> > > } >> > > >> > > ================ >> > > >> > > If I try running Kafka Connect stadnalone, the error message is the >> same. >> > > >> > > Has anybody faced this already ? What am I missing ? >> > > >> > > Many thanks to anybody reading this ! >> > > >> > > Jehan >> > > >> > >> > >
