#general
@prakashc: @prakashc has joined the channel
@vikash93s19: @vikash93s19 has joined the channel
@berkozbalci: @berkozbalci has joined the channel
@guillaume: @guillaume has joined the channel
@divcesar: I'm seeing what I believe are very large latency (a few hundreds MS) for processing some test queries in a local/test Pinot 0.5 setup and I'm wondering what might be the cause for that. Can anyone please advise if I'm doing something very wrong in this setup:
@mayanks: Can you share the broker response
@divcesar: Yeah. I'll get that. Just a sec
@divcesar: I updated the file in Dropbox. But here are some sample lines:
@divcesar: 2020/10/15 12:27:01.335 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-0] requestId=1,table=tpch_lineitem_OFFLINE,timeMs=615,docs=49313307/49313307,entries=0/49313307,segments(queried/processed/matched/consuming/unavailable):7/7/7/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=28,420,359,0,query=select sum(l_extendedprice) from tpch_lineitem LIMIT 1000 2020/10/15 12:27:52.097 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-7] requestId=8,table=tpch_lineitem_OFFLINE,timeMs=5,docs=1000/49313307,entries=0/1000,segments(queried/processed/matched/consuming/unavailable):7/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=1,3,8337,0,query=select l_extendedprice from tpch_lineitem LIMIT 1000 2020/10/15 12:28:16.374 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-13] requestId=14,table=tpch_lineitem_OFFLINE,timeMs=631,docs=49313307/49313307,entries=0/49313307,segments(queried/processed/matched/consuming/unavailable):7/7/7/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=1,629,8348,1,query=select l_extendedprice from tpch_lineitem order by l_extendedprice LIMIT 1000
@mayanks: Note, some of your queries are scanning the entire data: `49313307/49313307`
@divcesar: Yeah. So, should I expect that kind of latency for this scenario?
@mayanks: Some of the TPCH queries just do selection (no aggr, or group-by). While those queries are certainly supported, we have mainly focused on optimizing for aggr/group-by.
@mayanks: Depending on whether it is actually the CPU work-done, or there are other factors such as paging etc, you might be able to further reduce by playing around with JVM/GC settings
@divcesar: Thanks @mayanks! This is exactly the work I'm doing now. I.e., check whether there is a GC (memory) or CPU bottleneck
@divcesar: As a follow up about the use of aggregation operation in queries. If I send this query "select sum(l_extendedprice) from tpch_lineitem" it takes about 700ms
@divcesar: Do you think this dataset is too large for a single server?
@mayanks: What's the total segment size, and the JVM settings?
@divcesar: Dataset: TPCH - 7 Segments of about 400MB - About 5M rows
@divcesar: Pinot: 0.5 OpenJDK: 1.8.0_252 JVM Args: -Xms8G -Xmx8G -XX:MaxDirectMemorySize=4G -XX:+ManagementServer -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:log.gc
@mayanks: 400MB per segment?
@divcesar: I'm using JDK 1.8 because it's what the client uses.
@divcesar: Yeah, at least the on disk file is about 400MB
@mayanks: Let's move to <#C011C9JHN7R|troubleshooting> ?
#random
@prakashc: @prakashc has joined the channel
@vikash93s19: @vikash93s19 has joined the channel
@berkozbalci: @berkozbalci has joined the channel
@guillaume: @guillaume has joined the channel
#fraud
@guillaume: @guillaume has joined the channel
#troubleshooting
@hunterelbourn: @hunterelbourn has joined the channel
@elon.azoulay: Hi, we are upgrading to pinot 5 and wanted to change table schemas to use date time field instead of time field spec. Can we just post to schema endpoint or will we have to create new tables?
@npawar: there’s some checks during the update, which will prevent backward compatible changes. And changing the field type (from TIME to DATETIME) will be considered backward incompatible. As of now the options would be - create new table, or let it be as it is
@npawar: *backward incompatible
@elon.azoulay: Yep, thanks! This really helps:)
@guillaume: @guillaume has joined the channel
@mayanks:
@mayanks: @divcesar How many CPU cores?
@mayanks: Also, could you provide a bit more context on what is the goal here?
@divcesar: 16 Cores. 16GB RAM.
@divcesar: I'm doing some profiling to understand what's the bottleneck of the current setup - which AFAICT is 'similar' to the one used by a client.
@divcesar: I don't know if this latency that I'm observing is something that I should expect given the current setup of if I'm misconfiguring something
@mayanks: It does not seem out of the ballpark, but there's definitely room for improvement
@mayanks: BTW, IMO, if you have your client's setup (data/query), I think that would be more helpful in profiling/optimizing
@divcesar: With this setup if I setup a harness client to send 10 queries concurrently I start seeing couple of seconds of latency. I was really surprised by that.
@divcesar: Fully agree with that.. I'm trying to get that.
@mayanks: Yeah, we have various optimizations for different cases. For example in your case of `sum() on all records`, star-tree index would work perfectly (as it will pre-compute some of the cubes).
@mailtobuchi: @divcesar FWIW, I've noticee the max GC pause to be 200ms in your settings. I've recently done some optimization around that by setting it to lower value like 20ms and that has improved query latencies. It depends on overall GC patterns though. If you want to confirm, analyze GC logs
@divcesar: Thanks Buchi. I'll play with that.
#pinot-dev
@guillaume: @guillaume has joined the channel
#community
@guillaume: @guillaume has joined the channel
#announcements
@guillaume: @guillaume has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
