#general
@darren: @darren has joined the channel
@darren: Hiiiiii all . :+1: :heart: :slightly_smiling_face:
@fx19880617: hello
@gilli.aswin: @gilli.aswin has joined the channel
@hisham.itani: for those interested, there's a Pinot subreddit
@ho0001vu: @ho0001vu has joined the channel
@npawar: Hey folks, @chinmay.cerebro and @baliga from the community will be speaking at Flink Forward, which is happening today! They will talk about how Apache Pinot and Apache Flink power realtime analytics at Uber. Their session is at 12.50pm PDT. Please tune in if you’re around!
@brianjstafford: @brianjstafford has joined the channel
#random
@darren: @darren has joined the channel
@gilli.aswin: @gilli.aswin has joined the channel
@ho0001vu: @ho0001vu has joined the channel
@brianjstafford: @brianjstafford has joined the channel
#announcements
@muralidharanece: @muralidharanece has joined the channel
#metadata-push-api
@steotia: Hi, unrelated to the metadata push API and the recent discussion on table lock: Are there any thoughts on the following for current non-deepstore based segment push to controller: • getting rid of untarring business by doing a multi-part push comprising of segment data and metadata separately • controller has to untar everything, fish out metadata and then do CRC check etc. If the metadata is provided separately, it doesn't have to do anything. • Our localTemp space for controllers is on HDD and we are seeing 30MB/sec there
@steotia: @mayanks @fx19880617
@mayanks: We can add optional metadata along with payload
@mayanks: And use SSD
@steotia: exactly
@mayanks: Will that solve our problem
@steotia: SSD for sure if we are not able to get rid of untarring business
@fx19880617: I somehow feel the bottleneck is the controller download, if we can bypass this path, then controller overhead will be reduced a lot
@mayanks: Yes controller bypassing will help a lot
@mayanks: Unfortunately at lnkd we don’t have a reliable deepstore
@fx19880617: we can add an option to convert metadata.properties and creation.meta file to json and let controller to handle that as well
@fx19880617: basically there is no need for controller to download the data part
@fx19880617: even current just metadata tar file helps a lot on the controller side
@mayanks: But we don’t have a deepstore that is reliable and can be used in prod
@mayanks: So we have no way to bypass controller
@fx19880617: ic
@fx19880617: not even hdfs?
@fx19880617: pinot segments are anyway generated there and stored right?
@mayanks: Availability is not good enough for production
@fx19880617: ic
@fx19880617: even offline push is not good ?
@mayanks: Offline push is good
@fx19880617: for realtime side
@fx19880617: Uber has implemented the p2p download
@fx19880617: maybe you wanna try out
@mayanks: So we are trying out ideas for deepstore
@fx19880617: so servers can download from each other
@fx19880617: got it
@mayanks: But in case none of those work we want to still optimize with controller in the path
@fx19880617: maybe 2 hdfs :slightly_smiling_face:
@fx19880617: hdfs1:// and hdfs2://
@fx19880617: I remember at Uber they used to use two hdfs to satisfy the availability requirements :stuck_out_tongue:
@mayanks: :smiley:
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
