yswdqz commented on issue #9661: URL: https://github.com/apache/skywalking/issues/9661#issuecomment-1383703791
I have tested many ways, I'm sorry to say that for the current performance, it is difficult to meet APM's parsing needs for sql. I've thought about optimizing performance by simplifying parser rules, skipping unneeded parts, and using antlr4 actions. But there is no order of magnitude improvement. This is the performance report of my parser : [performance report](https://docs.google.com/document/d/1VlZv5yn43DCZWKMj-laQ3ftTyJDe8dQyggSFN8aiLY4/edit) This is the design report of sql-sign: [sql-sign](https://docs.google.com/document/d/1vcLNsT0OqJ-5CUfhi82-cEokSFQpe8sPyPxi85tGR1U/edit#heading=h.ce29jil1ejod) This is the repository of my parser: [sql-parser](https://github.com/yswdqz/sqlparser) Notice: it has 2 modes: with action and with visitor. With action mode can parse most dml statement. But visitor mode I only implement select and delete statement (because of after finish these, I already found that the problem of performance, so I have not completed it) To test them in APM , we can use [test file](https://github.com/yswdqz/sqlparser/blob/main/test/RealSegmentTest.java). It should be put in `server-starter` module's test folder. And change `VirtualDatabaseProcessor#prepareVSIfNecessary` as [VirtualDatabaseProcessor.java](https://github.com/yswdqz/sqlparser/blob/main/test/VirtualDatabaseProcessor.java) . Then you can run benchmark to test the performance changes before and after code changes. Finally, as mentioned in the performance report, The main performance problems are in the lexer analysis and parser analysis sections. So it's likely that future performance improvements will be in the parser and lexer. It might be a good idea not to use antlr4 (or other util) to implement the parser, but that would be too much work. In the future I will focus on solutions from other open source frameworks and try to solve this performance problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
