Hi all, In the last few months, I've reworked major internals of S3QL. Before I make a new release, I'd appreciate if some more users could give the current Git version a test and report any issues. This is particularly important for users of backends other than S3 and Google Storage (since I only run integration tests for those).
*For those interested, some details on the changes:* In terms of user-visible changes, it is now possible to separately specify the number of threads to use for compression/encryption, and the number of connections to use for concurrent uploads (previously these two numbers had to be identical). Behind the scenes, this was only possible because the previously awkwardly-entangled mess of multithreaded and async code has been fully cleaned up. Until now, S3QL had an async file-system operations layer that called a partially async, partially threaded block layer, which relied on a synchronous backend layer that awkwardly called an HTTP layer intended for async users. The reasons for this are mostly historic (S3QL was started before Python had proper async support). Now, however, all of S3QL is async all the way to the HTTP layer, and only compression/encryption is cleanly moved to separate worker threads. Apart from the benefits of a cleaner codebase, this will make it much easier to implement some long-planned features, most importantly metadata upload/download utilizing multiple threads and multiple connections. So, please give it a try and repart back any issues! Best, -Nikolaus -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/s3ql/a63380d2-e586-491b-b23c-d2a2eff9903e%40app.fastmail.com.
