#general
@saumya2700: Thanks @mayanks, I tried AvroSchemaToPinotSchema utility but it is giving error, ```Caused by: java.lang.IllegalStateException: Not one field in the RECORD schema at shaded.com.google.common.base.Preconditions.checkState``` this error doesn't make sense as schema is correct and working with our messages.
@mayanks: Are all the columns you’d like to read into Pinot nested?
@saumya2700: I just need one column where I want to dump whole message in json format. I am able to add different column for every nested object from message but I dont want to do that because our schema has many nested fields which has further fields in it. We want whole message as it is in one column, is it possible to do with transformationConfig. For now what I did like below: ```"ingestionConfig": { "transformConfigs": [ { "columnName": "header_json", "transformFunction": "jsonFormat(header)" }, { "columnName": "dact_json", "transformFunction": "jsonFormat(dact)" }, { "columnName": "ref_json", "transformFunction": "jsonFormat(refData)" } } ]``` Avro schema is like : ```{ "fields": [ { "name": "header", "type": { "fields": [ { "name": "fid", "type": "string" }, { "name": "timestamp", "type": "long" } ], "name": "ABCHeader", "type": "record" } }, { "name": "status", "type": { "name": "NNStatus", "namespace": "com.common", "symbols": [ "N200_SUCCESS", "N400_SERVER_ERR" ], "type": "enum" } }, { "name": "dact", "type": { "fields": [ { "name": "lid", "type": "string" }, { "name": "timestamp", "type": "long" } ], "name": "DactHeader", "type": "record" } }, { "name": "defId", "type": "string" }, { "name": "fixTimestamp", "type": "long" }, "name": "CaRecord", "namespace": "com.mbn", "type": "record" }```
@mayanks: For whole message, as Kishore mentioned, you could use JSON index. But it would be good to understand how you want to query that JSON.
@paularmand.verhaegen: @paularmand.verhaegen has joined the channel
@karinwolok1: :wave: Help us welcome all the new Apache Pinot members! :wine_glass: Please tell us who you are and what brought you here! :smiley: @paularmand.verhaegen @itay @yoav @adam.hutson @haitao @ysuo @puneet.singh @sdg.deep @andrzej.holowko @ameyacm @jain.chitrang @mndoping @az983wp @ghanta.vardhan @kalpeshsaubhri @hardik.joshi @padma @nrajendra @pkhobragade @osskalluri @mehtashailee21 @erwin.sapuay1 @sanat.pattanaik @jkinzel @srishb @shupadha
@ankitraj: @ankitraj has joined the channel
@et_tech: @et_tech has joined the channel
@osskalluri: Hi Gang, this is Raj from Austin TX. I am a Cloud and Streaming Data enthusiast. Wanted to learn more about Pinot, and felt this is a good spot to be in the know.
@mayanks: Welcome to the community @osskalluri. Please feel free to ask questions and share your thoughts.
@abhinav.wagle1: Hi All. Is there a tool actively used by the community for benchmarking query performance of Pinot Cluster?
@mayanks: There’s a `query-runner.sh` that is built using
@jreissup: @jreissup has joined the channel
@nizar.hejazi: Hey team, I see that support for storing decimals as byte[] and SumPrecision function was added in
#random
@paularmand.verhaegen: @paularmand.verhaegen has joined the channel
@ankitraj: @ankitraj has joined the channel
@et_tech: @et_tech has joined the channel
@jreissup: @jreissup has joined the channel
#troubleshooting
@paularmand.verhaegen: @paularmand.verhaegen has joined the channel
@diana.arnos: :wave: hey there I have a strange situation going on. I have 2 servers setup up. Eventually, they had a problem and restarted and still running fine.I can see in the logs that they are consuming data normally: ```Consumed 261 events from (rate:3.1030054/s), currentOffset=763096, numRowsConsumedSoFar=288096, numRowsIndexedSoFar=288096 .... [Consumer clientId=consumer-455, groupId=] Discovered group coordinator <redacted> (id: 2147483646 rack: null) ``` But the controller still show them with `dead` status and when I try to query the data, I see in the Broker log: ```No server found for request 1: select responseId from responseCount limit 1``` And this is the response from the query API: ```{ "exceptions": [], "numServersQueried": 0, "numServersResponded": 0, "numSegmentsQueried": 0, "numSegmentsProcessed": 0, "numSegmentsMatched": 0, "numConsumingSegmentsQueried": 0, "numDocsScanned": 0, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 0, "numGroupsLimitReached": false, "totalDocs": 0, "timeUsedMs": 0, "offlineThreadCpuTimeNs": 0, "realtimeThreadCpuTimeNs": 0, "segmentStatistics": [], "traceInfo": {}, "minConsumingFreshnessTimeMs": 0, "numRowsResultSet": 0 }``` How can I make the Controller see they are alive? :eyes:
@mayanks: Can you check the idealstate and externalview in ZK browser? If server shows up there then may be the broker routing table needs to be rebuild (swagger api).
@luisfernandez: `/tables/{tableName}/rebuildBrokerResourceFromHelixTags` that one right?
@diana.arnos: yep, the servers show up there. I tried that one, the response is ```{ "status": "Broker resource is not rebuilt because ideal state is the same for table: responseCount_REALTIME" }```
@walterddr: does idealstate matches externalview?
@mayanks: Yes, please check if idealstate matches externalview - You can do so manually, or via debug endpoint.
@ankitraj: @ankitraj has joined the channel
@et_tech: @et_tech has joined the channel
@jreissup: @jreissup has joined the channel
#getting-started
@paularmand.verhaegen: @paularmand.verhaegen has joined the channel
@ankitraj: @ankitraj has joined the channel
@et_tech: @et_tech has joined the channel
@jreissup: @jreissup has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
