Hello, I am faced with performance issue. My s3ql mount attached to s3 backend with options: --cachesize 20000000 --max-cache-entries 1000000 --threads 16 --allow-other --backend-options no-ssl,tcp-timeout=30 --nfs I copy into s3ql mount many small files 100 - 1000 bytes. With time copy speed slowdown almost stall. I see almost 16 connections with s3 servers, but tcpdump shows small rate of uploading PUTs. Ok, I remounted with --threads 1024 ( I bet it was a bad idea), but connections with s3 was 24 max and lots of cross locks between threads. 1. How to increase parallels request to s3? Or increase upload performance? 2. What limits could be reasonable for --threads parameter? Why? SQLite bottleneck?
Any success stories on handling many small files at s3ql? Thanks, -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
