Hi Cleanup after a run is a real gap, especially with a persistent backend, leaving "orphan" metadata. It also can leave your catalog half-configured, with stuff like missing roleArn, etc. So, I think it's a welcome new feature :)
If I agree it's faster to reset the persistence layer directly, I would keep the API-based approach as the default. The reason is that it works with any backend and keeps the benchmark a plain client of a running server, I don't think it would be a good idea for benchmark to manipulate directly the persistence layer (or at least it should be optional as it would depend of the persistence backend). About DROP_WITH_PURGE_ENABLED prerequisite, I think turning on that flag changes more than view drops. Every table drop becomes a purge, which deletes data in object storage. If someone points the simulation on the wrong catalog or a shared bucket, they could lose real data. I would suggest to limit delete objects to the simulation mode, and also requiring an explicit opt-in (let's not forget the documentation about that). I'm not sure I understand why dropping a view needs purge enabled? If that's a server-side weird stuff rather than intended behavior, it may be better to fix on the server rather than documented as prerequisite. Also, last comment: cleanup should be safe to re-run (again and again :)). If it fails partly, a second run should skip objects that are already gone and carry on. I will take a look on the PR as well. Regards JB On Thu, Oct 8, 2026 at 6:52 AM Yong Zheng <[email protected]> wrote: > Hello all, > > The current benchmark tool offers a list of simulations > (create/read/update) and it has the ability to skip creation as people make > change to the benchmark config (e.g. change from 15 tables to 20 tables, > only 5 new tables would be create on the second run of create dataset > simulation). > > There is one problem I am seeing with current tool is the lack of the > ability to perform cleanup after runs of benchmarks. When using > non-persisted backend, current tool is great. However, when we are actually > testing against a persistent backends (e.g. metadata and S3), lack of > ability to cleanup can make the re-run of benchmarks be in-accurate due to > larger metadata created on those tables after a set of simulations. Also, > if an operator put the wrong catalog properties and use the benchmark to > create dataset (which would create catalog), this will never move forward > (e.g. forgot to put roleArn for S3 catalog). In this case, the operator > will be responsible to do the catalog update via API or CLI tool or do the > cleanup manually then rerun with updated config. The lack of the ability to > perform cleanup will leave the cleanup tasks to the operators which add > some amount of overhead. > > I am thinking to add a delete simulation which does the opposite of create > dataset simulation (it will drop tables/views then ns and catalog itself). > The one catch I am seeing is polaris has DROP_WITH_PURGE_ENABLED default to > false while drop the view would need this to be true. Thus, I am thinking > just to document this change needed on server side before people can use > delete simulation. This also means, the drop table will be purge instead of > de-register. Thoughts? > > Thanks, > Yong Zheng >
