#general
@ashwinviswanath: @ashwinviswanath has joined the channel
@karinwolok1: :wave: Please help us welcome our newest community members :wave: (also, help us celebrate reaching 1,700 members :wine_glass:!! :partying_face: ) :arrow_right: New members, please tell us who you are and what brought you here! How did you stumble across Apache Pinot :pinot: ? @sina.tamizi @jmint @chandanchoudhary716 @aravindchoutpally @qiuxgm @mgjain @ashwinviswanath @itdev @samsarode89 @yogesh.dandawate @vutruongxuan99 @dadelcas @diwa414 @shish @richard892 @nishad.sadasivan @debabrata @michsre @leon.bam @srirams.ganesh @madhan.r @pulugurtha @joinstu12 @dikesh.shah @andruszd @ankitgupta4894 @singitamkr @andruszd @david.cyze @simone.franzini @deep @mitn.azn @mayur.bagwe @steve.reed
@dadelcas: I'm a software engineer and I knew about pinot whilst investigating platforms for a big data analytics project. Back then documentation was really scarce. I'm interested in open source and I'm looking at contributing to the project, I've opened my first PR few days ago and I'm waiting for feedback :) exiting times! Thanks for the welcome message Karin!
@karinwolok1: Nice! That's so awesome, Dan!!!! We're so happy to have you as part of the project!
@sina.tamizi: Hi All, I am a telecommunication engineer, working on various projects. I came across Pinot looking into realtime analytics platforms specifically around network monitoring and automation. Hoping to contribute to the open source project. Thanks Karin!
@gqian3: Hi team, does Pinot currently support pagination with group by using limit a,b; if not, is there any plan to support this?
@mayanks: Pinot does not support pagination in group by. No plan at the moment that I am aware of. Do you mind sharing the use case that requires pagination?
@ashish: Question on how group by aggregations are implemented in Pinot. From what I understand, groupKeys are generated from group by columns’s dictionary ids. How does this work for intersegment and inter server aggregations?
@mayanks: The dictionary id based grouping happens at segment level. Since column dictionary ids are segment specific, it cannot be used for server/broker level grouping. There we have to default to using actual values. May I ask what's your issue? Is it just for understanding of the system, or are you facing an issue that needs to be addressed?
#random
@ashwinviswanath: @ashwinviswanath has joined the channel
#troubleshooting
@ashwinviswanath: @ashwinviswanath has joined the channel
@wentjin: Hi team, I found that Pinot sometimes shows that the table status is bad, and at the same time the corresponding segment is also bad (one replicas state is consuming, and the other is online), this bad state will be automatically recovered. So when querying, especially on-demand will the query get inconsistent results, or how will Broker choose which replicas to query when querying?
@g.kishore: Are you seeing this in UI
@wentjin: This will be recovered soon. We hope to add some alarm items by monitoring the table state, but the bad state will always appear. I guess it is because the consumption speed of different replicas of the segment is slightly different. One replica has been submitted and the other is still consuming.
@wentjin: Here is the bad segment state.
@g.kishore: its a UI bug.. for a short second when the segment is transitioning from one state to another, it shows it as bad
@wentjin: This API ‘
@g.kishore: Can you see external view?
@wentjin: External view of this table is garbled while other tables are fine.
@mayanks: If you are using 0.8.0, can you check the debug endpoint?
@mayanks: Also, independently, there's an api (check swagger) on getting the uncompressed ideal-state/external-view
@mosiac: Hello team, does Pinot need any system dependencies to run unit tests?
@mayanks: Not really, are you referring to JVM or CPU/Mem or some libraries?]
@mosiac: Mostly libraries that should be installed in a docker container running the tests
@mayanks: Should not need anything additional.
@mosiac: Thank you!
@luisfernandez: hey, does anyone have any recommendations in a production env as to what to do around capacity for the pinot-server, I’m working on a proof of concept with some real time data, and i already filled up my disk space, what are some of the things we can do to mitigate increasing disk space. I have been reading about this stuff:
@mayanks: What's the per-day data size you expect to be stored on Pinot servers? And how much are they storing now?
@ssubrama: @luisfernandez back of the envelope storage requirement for a realtime-only table : `numReplcias * dataPerDay * retentionDays`. Hopefully you can get an approximate value of data per day via the size api we have on the controller and dividing it by the number of days your table has been in place. Unless your ingestion rate changes heavily this should hold at a high level
@luisfernandez: thank you, so my servers are really small so they have 4GB, and they consumed data for 7hours till they got the disk filled up so at this rate i may be looking at ~14GB per day of data? and I have 2 servers going.
@luisfernandez: if I configure a retention policy it means that after that retention is done I cannot longer access that data thru pinot yea?
@ssubrama: pinot will remove data that is older than retention time, and free up space. There is a background job in the controller (retention manager) that you need to make sure is enabled (it is enabled by default). Each time it runs, it removes the segments that are completely outside the retention window (looks at the time column value, not really the "age" of data).
@luisfernandez: that makes a lot of sense @ssubrama thank you very much, one last question, what are some of the strategies there are to save disk space, do we compress any of the data in pinot or anything of the sort? thank you very much
@ssubrama: pinot data is highly compressed (except for the currently consuming segment). It is always a space/performance trade-off. You can remove some indexes if you are willing to take additional query latency. We have a controller API on what may be most effective way of setting up your indexes, you should explore that.
@mayanks: @luisfernandez Just to ensure, even if you use tiered storage or managed offline flows, you do need Pinot servers with attached (network attached is fine) to have the servers host the data for your retention and replication.
@luisfernandez: i’m not sure i followed that last message Mayank
@luisfernandez: but it did bring a new question
@ssubrama: The segment store is the actual copy of pinot data. Consider the data in servers as "cached". The servers download all the segments they are supposed to "host". As of now, they cache all the segments they host until the segments are removed (or retained out), which effectively means a second copy is stored in the servers. In fact, if you have N replicas, then N additional copies are stored. There is some very early work in progress to age out segments from servers (but still keep them in deep store) and fetch them on demand, but this is in _very_ early stages, so, for now, you should account for N additional copies in the servers.
@abhijeet.kushe: we are trying to configure s3 as deep storage based on this document
@pavel: @pavel has left the channel
@abhijeet.kushe: The segments are not being copied to s3.I get this message ```Moved segment airlineStats_batch_2014-01-01_2014-01-01 from temp location /tmp/pinot-tmp-data/fileUploadTemp/tmp-66a57920-be10-41a0-a5e3-3f752b660d7c to /var/pinot/controller/data,
@luisfernandez: are you getting any errors in the logs?
@abhijeet.kushe: They are actually being copied to a location “/var/pinot/controller/data\,s3\:/ca-ctct-transient-dev-us-east-1-eigi-datalake/pinot-data/pinot-s3-example/controller-data/” instead of s3
@abhijeet.kushe: No errors
@luisfernandez: in this case you are configuring the controller to upload to s3 yea?
@abhijeet.kushe: ```controller: name: controller replicaCount: 1 podManagementPolicy: Parallel podSecurityContext: {} # fsGroup: 2000 securityContext: {} probes: endpoint: "/health" livenessEnabled: false readinessEnabled: false persistence: enabled: true accessMode: ReadWriteOnce size: 1G #mountPath:
@abhijeet.kushe: Yes this is the config
@abhijeet.kushe: server config ``` # Extra configs will be appended to pinot-server.conf file extra: configs: |- pinot.set.instance.id.to.hostname=true pinot.server.instance.realtime.alloc.offheap=true pinot.server.instance.currentDataTableVersion=2 pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-east-1 pinot.server.segment.fetcher.protocols=s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher```
@luisfernandez: ```controller.helix.cluster.name=pinot controller.port=9000 controller.data.dir=
@luisfernandez: this is what i have for controller, it’s gcs but should be the same thing overall
@luisfernandez: how often do you flush segments?
@abhijeet.kushe: This was actually batch ingestion job …the airline stats example
@abhijeet.kushe: which did not push to s3
@abhijeet.kushe: Not a part of the realtime but offline table
@abhijeet.kushe: can u share your server config
@luisfernandez: ```pinot.server.netty.port=8098 pinot.server.adminapi.port=8097 pinot.server.instance.dataDir=/var/pinot/server/data/index pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment pinot.set.instance.id.to.hostname=true pinot.server.instance.realtime.alloc.offheap=true pinot.server.instance.currentDataTableVersion=2
@luisfernandez: this is what i have for server
@luisfernandez: but i’m working with a realtime table, it does upload to gcs
@abhijeet.kushe: hmm did u try the airline stats example ?
@luisfernandez: i didn’t try that i just did it with custom data
@abhijeet.kushe: ok but u tried the batch ingestion thats what I was trying to imply
@luisfernandez: ohh lol yea i haven’t just realtime stuff at the moment
@luisfernandez: but it seems based on the architecture diagram that the job itself is the one that should upload to the segment store
@luisfernandez: and then have the server pull from s3
@abhijeet.kushe: ohh i will realtime table and see if that goes through
#s3-multiple-buckets
@ashwinviswanath: @ashwinviswanath has joined the channel
#getting-started
@ashwinviswanath: @ashwinviswanath has joined the channel
@sina.tamizi: @sina.tamizi has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
