zuston commented on PR #294: URL: https://github.com/apache/incubator-uniffle/pull/294#issuecomment-1327007799
> > > I think in most cases, we only need to read from one replica, so i think we can give priority to AQE. > > > > > 100w tasks will occupy 125k memory. > > > > > > > > > > > > And i think this is acceptable and `taskIdBitmap` is more precision for multiple replicas. @jerqi @zuston > > > > > > It's OK for me. I think we could disable this taskIdBitmap filter in no-AQE optimization. And especially for AQE, the taskIds size for one reader should not be large. > > > > > > What do u think so? @jerqi @xianjingfeng > > > > We should only pass the taskBitmap when the reduce task has skew data. Yes. Do you want to continue according to this solution? I think I could finish this today due to too much read pressure from AQE memory duplicate reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
