#general
@karinwolok1: :heart: Help us welcome our newest community members! :wave: :wine_glass: Would love to know who you are and what brought you here! :smile: @gonzalo @vineeth0025 @vipin.s.kumar @milo.venti84 @tokuda.sss @juanpabloalvis @wzcyc @gaetanmorlet @nkhelifaoui @davis @vicky301186(fun name! :rolling_on_the_floor_laughing: ) @rohit @zyuanyuz @avinash @sdshah1994 @raghavan.eshwar94 @nisheetkmr @jmodha @fritzb
@chxing: Hi All do we have some example to extract attributes from json objects in transformFunction
@npawar: you would typically use jsonFormat or jsonPath functions.
@chxing: Hi @npawar Thx for your replay, If json path is null, will pinot throw exception or just give null instead thx
@npawar: if your whole jsonField (e.g. person) is null, you’ll get an exception. But if just the jsonPath (person.address.state) is null, you will get null, or default value that you set ```JSONPATHSTRING(person, '$.address.state', 'CA')```
@npawar: are you seeing different behavior?
@chxing: Got it, thx @npawar
@chxing: I can’t find some examples in wiki, thx
@mrpringle: is there a way to control skipUpsert through the connection string. trying to use superset to get history but it doesn't allow us to edit generated pinot queries to add the option.
@g.kishore: Don’t think so..
#troubleshooting
@bajpai.arpita746462: Hello Everyone, I am trying to move data from REALTIME to OFFLINE table after schema evolution. I have added a new column age of type INT and since I did not provide any value it is reflecting 0 in the real-time table , but when the task RealtimeToOfflineSegmentsTask is getting executed in minion console it is throwing exception "INFO": "org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field age of record" if anyone has any idea regarding the same , please do share
@mayanks: @jackie.jxt ^^
@jackie.jxt: @bajpai.arpita746462 Can you please try with the latest master, or wait for the new release which should be available soon? This `null` handling issue is a known issue and should have been fixed
@bajpai.arpita746462:
@syedakram93: {"code":400,"error":"Invalid transform function
@syedakram93: facing above error while adding groovy function
@syedakram93: can someone help?
@syedakram93: "ingestionConfig": { "transformConfigs": [{ "columnName": "visits_case", "transformFunction": "Groovy((CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end), visits_case)" } }
@npawar: Syntax looks incorrect. Curly brackets are needed for function:
@npawar: Also you're using first_tab but it's not being passed as argument
@syedakram93: U mean in transformation function, shall I pass both visits and first_tab as well ?
@npawar: Is first tab one of the Pinot columns? Is yes, then yes you need to pass both
@syedakram93: {"code":400,"error":"Invalid transform function 'Groovy((CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end), visits, first_tab)' for column 'visits_case'"}
@syedakram93: i m getting exception still
@npawar: The syntax is still incorrect, look at the doc I sent
@npawar: Eg Groovy({arg1+arg2}, arg1, arg2)
@syedakram93: "ingestionConfig": { "transformConfigs": [{ "columnName": "visits_case", "transformFunction": "Groovy({(CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end)}, visits, first_tab)" }, { "columnName": "isbounce_case", "transformFunction": "Groovy({(case when isbounce = 'true' then 1 else 0 end)}, isbounce)" }, { "columnName": "isunique_case", "transformFunction": "Groovy({(case when isunique = 'true' then 1 else 0 end)}, isunique)" }, { "columnName": "isendandisbounce_case", "transformFunction": "Groovy({(case when isbounce = 'true' and eventnumber = 1 then 1 else 0 end)}, isend,isbounce)" }, { "columnName": "isend_case", "transformFunction": "Groovy({(case when isend = 'true' then 1 else 0 end)}, isend)" }, { "columnName": "eventnumber_case", "transformFunction": "Groovy({(case when eventnumber = 1 then 1 else 0 end)}, eventnumber)" }] },
@syedakram93: is this fine?
@syedakram93: not sure, how to include CASE inside groovy function
@syedakram93: fixed it
@syedakram93: thanks
@syedakram93: working
@npawar: awesome
@syedakram93: response: {"code":400,"error":"Invalid transform function 'Groovy((CASE WHEN visits = 1 and (first_tab = 'true' or first_tab = 'null') then 1 else 0 end), visitscase)' for column 'visitscase'"}
#transform-functions
@chxing: @chxing has joined the channel
#getting-started
@kangren.chia: hello, im just curious how queries run faster when you call the exact same query the second time round. how does caching work in pinot? if it matters, i’m using the inverted index, with no partition pruning (only 1 broker running). broker looks something like this: ```Processed requestId=34,table=sorted_events_OFFLINE,segments(queried/processed/matched/consuming)=198/198/198/-1,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=426,resSerMs=0,totalTimeMs=426,minConsumingFreshnessMs=-1,broker=Broker_172.26.0.4_8099,numDocsScanned=259467,scanInFilter=619119085,scanPostFilter=259467,sched=fcfs Slow query: request handler processing time: 427, send response latency: 3, total time to handle request: 430 Processed requestId=35,table=events_OFFLINE,segments(queried/processed/matched/consuming)=198/198/118/-1,schedulerWaitMs=0,reqDeserMs=5,totalExecMs=221,resSerMs=0,totalTimeMs=226,minConsumingFreshnessMs=-1,broker=Broker_172.26.0.4_8099,numDocsScanned=657,scanInFilter=346815,scanPostFilter=657,sched=fcfs``` also, i’m wondering what is considered prompts `"Slow query: …"` to show up in logs? does this mean pinot is suggesting that some optimization is possible to speed up my queries?
@mayanks: It is not caching, but cold start. Pinot mmaps index files, the OS needs to pull in pages the very first time.
@kangren.chia: understood. thanks!
@mayanks: Right now the default value to print `slow query` is when latency is `> 100ms`
@kangren.chia: so its a hardcoded value. got it! also kind of mindblowing that you guys chose 100ms as the hardcoded value :joy:
@mayanks: yeah, 100ms is `slow` for Pinot :wink:
#pinot-docsrus
@amrish.k.lal: So I updated the SUMMARY.md file, but the information is still not showing up in
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
