Hi Pierre, we managed to fix the issue using NiFi version 1.14. As we already planned to upgrade in near future, we are fine with that.
Thanks, Christian Von: Pierre Villard <[email protected]> Gesendet: Mittwoch, 27. Oktober 2021 20:29 An: [email protected] Betreff: Re: Unable to evolve Schema with HWXRegistry Thanks Christian. Understood, I'll try to have a look into the code and get back to you. Le mar. 26 oct. 2021 à 19:16, Weiss, Christian <[email protected]<mailto:[email protected]>> a écrit : Hi Pierre, the Avro-Files, which we write to a Kafka Topic in the next step, needs to be compatible with the Confluent Avro-Format, because we use Confluent compatible deserializers on the consuming end. Therefore we need to use the “Confluent Schema” Write-Strategy. The Hortonworks registry provides Confluent compatible Endpoints to achieving this. We are actually using NiFi 11.4. Hope this helps. Thanks, Christian Christian Weiß Data Architect Big Data/Analytics & IOT SVA System Vertrieb Alexander GmbH Freiberger Str. 39 01067 Dresden Mobile: +49 151 18027982 E-Mail: [email protected]<mailto:[email protected]> http://www.sva.de Geschäftsführung: Philipp Alexander, Sven Eichelbaum Sitz der Gesellschaft: Wiesbaden Registergericht: Amtsgericht Wiesbaden, HRB 10315 ________________________________ Von: Pierre Villard <[email protected]<mailto:[email protected]>> Gesendet: Tuesday, October 26, 2021 7:03:36 PM An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: Re: Unable to evolve Schema with HWXRegistry Hi Christian, Not sure to understand the overall objective. You're writing the schema reference using Confluent encoding but you have the schema in an Hortonworks Schema Registry instance? Also, what version of NiFi are you using? Thanks, Pierre Le mar. 26 oct. 2021 à 15:46, Weiss, Christian <[email protected]<mailto:[email protected]>> a écrit : Hey guys, we're facing some issues while writing avro with a connected hortonworks schema registry. We would like to achieve the follwoing: - registered a schema and evolve it over time - write a avro as "confluent avro" with magic byte zero followed by the schema id Therefor we've configured an AvroRecordSetWriter and a HortonworksSchemaRegistry as conroller service, so that it should retrieve the latest version. [cid:[email protected]] This works, unless we increase the version of a schema in the registry. We've investigate the requests, responses and the avro file itself and figured out the following. Actual behaviour: - The avro file contains the id which is present in the schemaMetadataId Expected behaviour: - The avro file contains the id which is present in the id field of the response Example call: curl -X GET "https://<host>:<port>/schema-registry/api/v1/schemaregistry/schemas/<schema_name>/versions/latest?branch=MASTER<https://%3chost%3e:%3cport%3e/schema-registry/api/v1/schemaregistry/schemas/%3cschema_name%3e/versions/latest?branch=MASTER>" -H "accept: application/json" Example response: { "id": 10, "schemaMetadataId": 7, "name": "schema_name", "description": "test", "version": 1, "schemaText": "..", "timestamp": 1635252068173, "stateId": 5, "mergeInfo": null } This makes it impossible to increase the version of an existing schema, because then the id and schemaMetadataId will evolve differently. Does anyone have similar issues? Thanks and cheers, Christian Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet. Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an [email protected]<mailto:[email protected]>. The SVA mail system is equipped with an email encryption gateway. If you want email sent to you to be encrypted please send a S/MIME signed email or your PGP public key to [email protected]<mailto:[email protected]>.
smime.p7s
Description: S/MIME cryptographic signature
