massdosage commented on pull request #1243: URL: https://github.com/apache/iceberg/pull/1243#issuecomment-669772790
> > > > `HiveRunner` is rather slow (I believe each test sets up a new local HMS from scratch) and `TestHiveIcebergInputFormat` takes about 4 minutes to run on my machine (compile time included). > > > > > > > > > In most of our test cases we spin up HMS once per test file and clean it up in setUp/tearDown. Not an ideal solution, but saves as time/money. > > > > > > HiveRunner doesn't currently support that but it has been discussed before - [klarna/HiveRunner#69](https://github.com/klarna/HiveRunner/issues/69) > > Hive does this here: > https://github.com/apache/hive/blob/6267520a50ba8c12ae5cee8fb27a6746376a21c6/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L444 > > I think the only part we are interested in here is: > https://github.com/apache/hive/blob/6267520a50ba8c12ae5cee8fb27a6746376a21c6/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L336 > > Which is very similar than what was done in [klarna/HiveRunner#69](https://github.com/klarna/HiveRunner/issues/69). We manually went through and deleted the tables in the databases (I think there was some issue previously which should be fixed now). The only other difference is that we clean up the files as well which is needed for EXTERNAL tables. Since the Iceberg tables will be EXTERNAL, I think we will need that as well here. That sounds promising. I guess the best place to do this is in HiveRunner itself. That issue referenced above has a PR against it from one of the original committers on HiveRunner but he makes it clear that it's very experimental and it hasn't been worked on in a long time. We could either build on that or raise a new one. I'd be happy to work with you on it if you want? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
