Hello all, The current benchmark tool offers a list of simulations (create/read/update) and it has the ability to skip creation as people make change to the benchmark config (e.g. change from 15 tables to 20 tables, only 5 new tables would be create on the second run of create dataset simulation).
There is one problem I am seeing with current tool is the lack of the ability to perform cleanup after runs of benchmarks. When using non-persisted backend, current tool is great. However, when we are actually testing against a persistent backends (e.g. metadata and S3), lack of ability to cleanup can make the re-run of benchmarks be in-accurate due to larger metadata created on those tables after a set of simulations. Also, if an operator put the wrong catalog properties and use the benchmark to create dataset (which would create catalog), this will never move forward (e.g. forgot to put roleArn for S3 catalog). In this case, the operator will be responsible to do the catalog update via API or CLI tool or do the cleanup manually then rerun with updated config. The lack of the ability to perform cleanup will leave the cleanup tasks to the operators which add some amount of overhead. I am thinking to add a delete simulation which does the opposite of create dataset simulation (it will drop tables/views then ns and catalog itself). The one catch I am seeing is polaris has DROP_WITH_PURGE_ENABLED default to false while drop the view would need this to be true. Thus, I am thinking just to document this change needed on server side before people can use delete simulation. This also means, the drop table will be purge instead of de-register. Thoughts? Thanks, Yong Zheng
