JkSelf commented on issue #4836: URL: https://github.com/apache/incubator-gluten/issues/4836#issuecomment-2041285118
@ayushi-agarwal Yes, Method 1 cannot solve the PartialWindow issue; not only is there no performance improvement, but it also fails to address the out-of-memory (oom) problem for Q67 under tight memory conditions. We need to adopt Method 2 to map the RankLimit operator in Spark to the TopNRowNumber operator in Velox. Currently, there is an issue that the documentation for TopNRowNumber states it only supports the row_number function. However, from looking at the code, it seems that it should also be able to support rank and dense_rank. I have added a test in Velox's unit tests to verify that rank and dense_rank can work using TopNRowNumber. I will confirm this with the Velox community later. Could you first support the mapping of RankLimit to Velox's TopNRowNumber in Gluten? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
