Query Avro based Kafka Topic using drill

[Jaiswal, Vibhor]

Hi Drill Support ,

I have been trying to query an avro based topic using drill . I am able to 
connect to topic and do a count query . But when I do a select on topic it 
gives error .

Does Drill allow querying  avro message for kafka or its only limited to json ?

Below is my storage plugin configuration –

{
  "type": "kafka",
  "kafkaConsumerProps": {
    "key.deserializer": 
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
    "sasl.mechanism": "SCRAM-SHA-256",
    "bootstrap.servers": "##############",
    "group.id": "drill-consumer",
    "value.deserializer": 
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
    "sasl.jaas.config": 
"org.apache.kafka.common.security.scram.ScramLoginModule required 
username=\"#####\" password=\"$$$$$$$\";",
    "session.timeout.ms": "30000",
    "security.protocol": "SASL_PLAINTEXT"
  },
  "enabled": true
}


Below is the exception –

MessageReader org.apache.drill.exec.store.kafka.decoders.JsonMessageReader 
DATA_READ ERROR: Failed to read JSON record
�Full64X155351Q4X SIGLB
aw89039934998H16[1]934998H16GENERIC_CUSIP[1]SMCP[1]479133318(INTEREST-RATE-FUTURE(LONG-INTEREST-FUTURETsy
 Future8ULTRA LONG-TERM UST BOND 
H21[1]‑SIGLB.934998H16��Q��ך��]�����Q�[1]��Q�[1]��Q�[1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1]324514956j�@[1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1]@cef047c10c3d4efd994d16dd533e5247[1][1]:
 Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) 
is allowed between tokens at [Source: (String)"
�Full64X155351Q4X SIGLB
aw89039934998H16[1]934998H16GENERIC_CUSIP[1]SMCP[1]479133318(INTEREST-RATE-FUTURE(LONG-INTEREST-FUTURETsy
 Future8ULTRA LONG-TERM UST BOND 
H21[1]‑SIGLB.934998H16��Q��ך��]�����Q�[1]��Q�[1]��Q�[1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1][1]324514956j�@[1][1]"[truncated
 158 chars]; line: 1, column: 2] MessageReader 
org.apache.drill.exec.store.kafka.decoders.JsonMessageReader [Error Id: 
19a85d55-7bee-4e34-9af9-df26fdc34340 ] Fragment 1:0 [Error Id: 
19a85d55-7bee-4e34-9af9-df26fdc34340 on sd-cba5-d6e8.nam.nsroot.net:31010]

Regards,
Vibhor Jaiswal
[Picture1]
• 212-657-1263 | • 862-222-2413