anthonyikeda opened a new issue, #20022: URL: https://github.com/apache/pulsar/issues/20022
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version 2.11.0 ### Minimal reproduce step ## Method ### Configure Schema Validation By default, Pulsar does not validate schemas, this needs to be reconfigured: conf/standalone.conf ```properties isSchemaValidationEnforced=true ``` ### Start Pulsar in Standalone mode Starting in the `PULSAR_HOME` directory: ```bash $ bin/pulsar standalone ``` ### Create our Tenant and Namespace Starting in the `PULSAR_HOME` directory: ```bash $ bin/pulsar-admin tenants create my-tenant $ bin/pulsar-admin tenants list my-tenant public pulsar $ bin/pulsar-admin namespaces create --tenant my-namespace $ bin/pulsar-admin namespaces list my-tenant my-tenant/my-namespace ``` ### Create our Topic ```bash $ bin/pulsar-admin topics create persistent://my-tenant/my-namespace/basic-topic-3 $ bin/pulsar-admin topics list my-tenant/my-namespace persistent://my-tenant/my-namespace/basic-topic-3 ``` ### Apply a schema to the topic Product Schema for AVRO ```json { "type" : "record", "namespace" : "com.pulsar.workshop", "name" : "Product", "fields" : [ { "name" : "product_id", "type" : "int" }, { "name" : "product_name", "type" : "string" }, { "name" : "product_description", "type" : [ "null","string" ] } ] } ``` ### Apply the schema to the topic topic-schema.json ```json { "type" : "JSON", "schema" : "{ \"type\" : \"record\", \"namespace\" : \"com.pulsar.workshop\", \"name\" : \"Product\", \"fields\" : [ { \"name\" : \"product_id\", \"type\" : \"int\" }, { \"name\" : \"product_name\", \"type\" : \"string\" }, {\"name\" : \"product_description\", \"type\" : [ \"null\",\"string\" ] } ] }" } ``` ```bash $ bin/pulsar-admin schemas upload -f topic-schema.json persistent://my-tenant/my-namespace/basic-topic-3 ``` ### Verify the schema ```bash $ bin/pulsar-admin schemas get persistent://my-tenant/my-namespace/basic-topic-3 { "version": 0, "schemaInfo": { "name": "basic-topic", "schema": { "type": "record", "namespace": "com.pulsar.workshop", "name": "Product", "fields": [ { "name": "product_id", "type": "int" }, { "name": "product_name", "type": "string" }, { "name": "product_description", "type": [ "null", "string" ] } ] }, "type": "JSON", "timestamp": 1680715759782, "properties": {} } } ``` ### Create a consumer to our Topic ```bash $ bin/pulsar-client consume persistent://my-tenant/my-namespace/basic-topic-3 -s 'my-subscription' -p Earliest -n 0 2023-04-05T09:37:36,229-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://my-tenant/my-namespace/basic-topic][my-subscription] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0 ``` ### Produce Messages apple.json ```json { "product_id" : 12, "product_name" : "Apple", "product_description" : "A red fruit" } ``` #### Standard produce expected to pass Pulsar Producer ```bash $ bin/pulsar-client produce persistent://my-tenant/my-namespace/basic-topic-3 -f apple.json -vs 'json:{ "type" : "record", "namespace" : "com.pulsar.workshop", "name" : "Product", "fields" : [ { "name" : "product_id", "type" : "int" }, { "name" : "product_name", "type" : "string" }, {"name" : "product_description", "type" : [ "null","string" ] } ] }' 2023-04-05T11:41:53,902-0700 [main] WARN io.netty.resolver.dns.DnsServerAddressStreamProviders - Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos' 2023-04-05T11:41:54,113-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x86c56ff2, L:/127.0.0.1:51950 - R:localhost/127.0.0.1:6650]] Connected to server 2023-04-05T11:41:54,195-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {"topicName":"persistent://my-tenant/my-namespace/basic-topic-3","producerName":null,"sendTimeoutMs":30000,"blockIfQueueFull":false,"maxPendingMessages":1000,"maxPendingMessagesAcrossPartitions":50000,"messageRoutingMode":"RoundRobinPartition","hashingScheme":"JavaStringHash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":1000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"chunkMaxMessageSize":-1,"compressionType":"NONE","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{},"initialSubscriptionName":null} 2023-04-05T11:41:54,213-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://localhost:6650/","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"connectionMaxIdleSeconds":180,"useTcpNoDelay":true,"useTls":false,"tlsKeyFilePath":"","tlsCertificateFilePath":"","tlsTrustCertsFilePath":"","tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsKeyStoreType":"JKS" ,"tlsKeyStorePath":"","tlsKeyStorePassword":"*****","tlsTrustStoreType":"JKS","tlsTrustStorePath":"","tlsTrustStorePassword":"*****","tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null} 2023-04-05T11:41:54,221-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [null] Creating producer on cnx [id: 0x86c56ff2, L:/127.0.0.1:51950 - R:localhost/127.0.0.1:6650] 2023-04-05T11:41:54,231-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-13] Created producer on cnx [id: 0x86c56ff2, L:/127.0.0.1:51950 - R:localhost/127.0.0.1:6650] 2023-04-05T11:41:54,269-0700 [main] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-13] Pending messages: 0 --- Publish throughput: 18.53 msg/s --- 0.01 Mbit/s --- Latency: med: 25.000 ms - 95pct: 25.000 ms - 99pct: 25.000 ms - 99.9pct: 25.000 ms - max: 25.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 99.000 bytes - 95pct: 99.000 bytes - 99pct: 99.000 bytes - 99.9pct: 99.000 bytes - max: 99.000 bytes --- Ack received rate: 18.53 ack/s --- Failed messages: 0 --- Pending messages: 0 2023-04-05T11:41:54,272-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-13] Closed Producer 2023-04-05T11:41:54,273-0700 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://localhost:6650/ 2023-04-05T11:41:54,278-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x86c56ff2, L:/127.0.0.1:51950 ! R:localhost/127.0.0.1:6650] Disconnected 2023-04-05T11:41:56,359-0700 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced ``` #### Standard produce expected to fail Pulsar Producer ```bash $ bin/pulsar-client produce persistent://my-tenant/my-namespace/basic-topic-3 -m '{"family" : "genus" }' -vs 'json:{ "type" : "record", "namespace" : "com.pulsar.workshop", "name" : "Product", "fields" : [ { "name" : "product_id", "type" : "int" }, { "name" : "product_name", "type" : "string" }, {"name" : "product_description", "type" : [ "null","string" ] } ] }' 2023-04-05T11:43:15,431-0700 [main] WARN io.netty.resolver.dns.DnsServerAddressStreamProviders - Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos' 2023-04-05T11:43:15,628-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x7bf98cc5, L:/127.0.0.1:51959 - R:localhost/127.0.0.1:6650]] Connected to server 2023-04-05T11:43:15,705-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {"topicName":"persistent://my-tenant/my-namespace/basic-topic-3","producerName":null,"sendTimeoutMs":30000,"blockIfQueueFull":false,"maxPendingMessages":1000,"maxPendingMessagesAcrossPartitions":50000,"messageRoutingMode":"RoundRobinPartition","hashingScheme":"JavaStringHash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":1000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"chunkMaxMessageSize":-1,"compressionType":"NONE","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{},"initialSubscriptionName":null} 2023-04-05T11:43:15,722-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://localhost:6650/","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"connectionMaxIdleSeconds":180,"useTcpNoDelay":true,"useTls":false,"tlsKeyFilePath":"","tlsCertificateFilePath":"","tlsTrustCertsFilePath":"","tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsKeyStoreType":"JKS" ,"tlsKeyStorePath":"","tlsKeyStorePassword":"*****","tlsTrustStoreType":"JKS","tlsTrustStorePath":"","tlsTrustStorePassword":"*****","tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null} 2023-04-05T11:43:15,729-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [null] Creating producer on cnx [id: 0x7bf98cc5, L:/127.0.0.1:51959 - R:localhost/127.0.0.1:6650] 2023-04-05T11:43:15,739-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-14] Created producer on cnx [id: 0x7bf98cc5, L:/127.0.0.1:51959 - R:localhost/127.0.0.1:6650] 2023-04-05T11:43:15,775-0700 [main] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-14] Pending messages: 0 --- Publish throughput: 19.44 msg/s --- 0.00 Mbit/s --- Latency: med: 25.000 ms - 95pct: 25.000 ms - 99pct: 25.000 ms - 99.9pct: 25.000 ms - max: 25.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 21.000 bytes - 95pct: 21.000 bytes - 99pct: 21.000 bytes - 99.9pct: 21.000 bytes - max: 21.000 bytes --- Ack received rate: 19.44 ack/s --- Failed messages: 0 --- Pending messages: 0 2023-04-05T11:43:15,778-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://my-tenant/my-namespace/basic-topic-3] [standalone-51-14] Closed Producer 2023-04-05T11:43:15,779-0700 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://localhost:6650/ 2023-04-05T11:43:15,783-0700 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x7bf98cc5, L:/127.0.0.1:51959 ! R:localhost/127.0.0.1:6650] Disconnected 2023-04-05T11:43:17,857-0700 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced ``` ### What did you expect to see? In the failed message produce, the data `-m '{"family" : "genus" }'` should have been rejected, instead it was accepted and sent to the consumers. ### What did you see instead? The message was accepted and published to the consumers of the topic. ### Anything else? So far I've only tried this in standalone mode with the `bin/pulsar-client` I've also tried configuring schema validation at the namespace level with no such luck: ```bash $ bin/pulsar-admin namespaces set-schema-validation-enforce --enable my-tenant/my-namespace $ bin/pulsar-admin namespaces get-schema-validation-enforce my-tenant/my-namespace true $ bin/pulsar-admin topics set-schema-validation-enforce --enable persistent://my-tenant/my-namespace/basic-topic-3 $ bin/pulsar-admin topics get-schema-validation-enforce persistent://my-tenant/my-namespace/basic-topic-3 true ``` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
