mxtymoshyk opened a new pull request, #41222: URL: https://github.com/apache/beam/pull/41222
Fixes #26495 `MongoDbIO.read().withBucketAuto(true)` fails on large collections with: ``` Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. ``` To split the read, the connector runs a `$bucketAuto` aggregation over `_id`. That stage sorts the whole collection, and MongoDB limits an in-memory sort to 100 MB. Servers older than 6.0 only write the sort to disk when the request sets `allowDiskUse`, and the Java connector never set it. The Python connector already sets it, and the Go connector added it in #25013. This PR sets `allowDiskUse(true)` in two places: - `BoundedMongoDbSource.buildAutoBuckets`, which computes the splits. Reads that use `withQueryFn(AggregationQuery...)` call it too, even without `withBucketAuto`. - `AggregationQuery.apply`, which runs the user's own pipeline. A large `$sort` or `$group` there hits the same limit. MongoDB 6.0 and later allow disk use by default (`allowDiskUseByDefault=true`), so nothing changes on those servers unless an admin turned that default off. In that case Beam reads now spill to disk anyway, because the connector sets the flag on every aggregation, as the Python and Go connectors do. I can add a `withAllowDiskUse(boolean)` option if you'd rather let the server setting win. Atlas Free and Flex clusters ignore `allowDiskUse` instead of rejecting it ([Atlas docs](https://www.mongodb.com/docs/atlas/unsupported-commands/)), so reads there behave as before. The error stays on those tiers for large collections, because they cap in-memory sorts at 32 MB and never spill to disk. Testing: - Two new tests in `MongoDbIOTest` record the commands the driver sends and check that the `aggregate` command carries `allowDiskUse: true`. Both fail without the fix. The server hardcodes the 100 MB limit for `$bucketAuto`, so a small test collection can't trigger the real error. - I ran the reporter's pipeline against mongo 4.4.17 with 150k documents (about 146 MB). It failed with the error above before the fix and read all 150,000 documents after it. - `./gradlew :sdks:java:io:mongodb:test` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
