#general
@naga.b: Hi All , Currently I am working in Pinot POC to solve OLAP based needs in our org.Our architect team worried about using zookeeper/helix as metadata storage . Lets say , if want to run Pinot for 5 year in production, at certain point of time ,zookeeper will run into disk space full issue as it store all segments metadata in it. right ? . [Druid is using sql database (mysql/postgresdb) to store this kind of metadata] Please suggest right solution to solve this problem. also curious to know about how other big giant (Uber ,Linkedin , etc ) using pinot in their production setup.
@g.kishore: Helix/zookeeper has been used in production for 10+ years. managing thousands of nodes. what Pinot stores in ZK is minimal and has never been an issue.
@mayanks: Yes, this is true for multiple orgs across different companies ^^ @naga.b
@ryantle1028: Hi ALL, I'm in POC Pinot. Pinot support real time table with kafka SASL_SSL (authen kerbelos + cert ) ?
@mayanks: @slack1
#troubleshooting
@ryantle1028: hi anyone can show me example config realtime table with kafka sasl_ssl.
@xiangfu0:
@ryantle1028: this is my table code { "tableName": "user_stream_kkts_test", "tableType": "REALTIME", "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName": "timestampInEpoch", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig": null, "crypterClassName": null, "peerSegmentDownloadScheme": null }, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [], "sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null, "aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "user_stream", "stream.kafka.broker.list": "kkts01:9094,kkts02:9094,kkts03:9094", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M", "stream.kafka.consumer.prop.group.id": "rdp_lookup", "security.protocol": "SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name": "bigfoot", "ssl.truststore.location": "/data/apache-pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password": "godzilla007", "sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true storeKey=true keyTab=/data/apache-pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab principal=kfuat_rdp@KAFKA.SECURE;" } }, "metadata": {}, "ingestionConfig": { "filterConfig": null, "transformConfigs": null }, "quota": { "storage": null, "maxQueriesPerSecond": null }, "task": null, "routing": { "segmentPrunerTypes": null, "instanceSelectorType": null }, "query": { "timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null, "tierConfigs": null }
@ryantle1028: ```{ "_code": 500, "_error": "org.apache.kafka.common.KafkaException: Failed to construct kafka consumer" }```
@ryantle1028: @xiangfu0 please help
@xiangfu0: does this config work with Kafka console consumer ?
@xiangfu0: If you can consume with Kafka console consumer config file, then you should be able to use same configs to ingest from Pinot
#flink-pinot-connector
@naga.b: @naga.b has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
