andrewmusselman opened a new issue, #31: URL: https://github.com/apache/tooling-gofannon/issues/31
### Summary The default loop fallback works for DynamoDB but doesn't take advantage of `BatchWriteItem` (25/req chunking) and `BatchGetItem` (100/req chunking). DynamoDB users don't get the bulk-API performance wins that CouchDB users get from `_bulk_docs` / `_all_docs?keys=`. Filed upstream at https://github.com/The-AI-Alliance/gofannon/issues/580 ### Details **Context:** Filed alongside the DB perf commit (S3 in the shipped list — bulk ops, optimistic writes, deferred access tracking). The framework supports bulk operations as a primitive; backends declare their implementation. CouchDB uses native bulk APIs; DynamoDB falls back to per-key loops. **Impact:** - DynamoDB-backed deployments don't see the bulk-op throughput improvements from S3. - Per-key requests against DynamoDB are slower and more expensive (per-request cost) than batched ones. ### Remediation Implement bulk methods in `services/database_service/dynamodb.py`: - `batch_get_many` using `BatchGetItem` (100/req chunking) - `batch_save_many` using `BatchWriteItem` (25/req chunking) - Retry `UnprocessedItems` from the response (DynamoDB returns partial results when throughput is exceeded) Reference commit: `3d88b234c06d2c977f8ca4bb043b529d21f4e9d1`. ### Acceptance Criteria - [ ] Fixed: `BatchGetItem` implementation with 100-key chunking - [ ] Fixed: `BatchWriteItem` implementation with 25-item chunking - [ ] Fixed: `UnprocessedItems` retry loop - [ ] Test added: 1000-item batch operations work correctly - [ ] Test added: Partial failures (some `UnprocessedItems`) handled - [ ] Benchmark: throughput comparison vs. existing per-key loop ### References - File: `webapp/packages/api/user-service/services/database_service/dynamodb.py` - Commit: `3d88b234c06d2c977f8ca4bb043b529d21f4e9d1` - Tracker: FIXES.md item #10 ### Priority **Low** - Already filed. DynamoDB users see degraded performance vs. CouchDB users until implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
