SEPURI-SAI-KRISHNA opened a new issue, #13330:
URL: https://github.com/apache/seatunnel/issues/13330

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22)
 and found no similar issues.
   
   ### What happened
   
   The MongoDB source splits a collection with `SamplingSplitStrategy`: it 
samples documents sorted by `partition.split-key` (default `_id`) and turns 
every few sampled values into the bound of a range split, `{key: {$lt: b1}}`, 
`{key: {$gte: b1, $lt: b2}}`, ..., `{key: {$gte: bn}}` 
(`MongoSplitUtils.createMongoSplit`). Once a collection is larger than one 
split (64 MB by default), the job can lose documents or read them more than 
once, and it still succeeds:
   
   1. **Documents are lost when the split key has more than one BSON type.** A 
MongoDB range filter only matches values of the type bracket of its bound 
(numbers, strings, ObjectIds, dates and so on). When two neighbouring bounds 
have different types, for example `{_id: {$gte: 760, $lt: "s0155"}}`, the split 
matches nothing, so every number above 760 and every string below `"s0155"` is 
never read. This happens with the default `_id` when a collection mixes 
ObjectId, string and numeric ids, which is common after imports.
   2. **Documents are read again when the split key is missing or null in some 
documents.** `samples.get(i).get(splitKey)` returns Java `null` for a sampled 
document without the key, and `createMongoSplit` treats a null bound as "no 
bound": a split between two such bounds is `{}` and reads the whole collection. 
A null or NaN value can also become a bound, `{key: {$lt: null}}` or `{key: 
{$lt: NaN}}`, of a range that matches nothing.
   3. **`match.query` is dropped on the first and last split when it names the 
split key.** `createMongoSplit` copies the match query into the split query and 
then appends the bound with the same field name, which replaces the condition 
of the match query.
   4. **A split key with a dot, such as `a.b`, makes every split `{}`.** 
`BsonDocument.get("a.b")` does not follow the path, so every bound is null and 
each split reads the whole collection.
   
   Measured with the connector's own `SamplingSplitStrategy` on MongoDB 8.0.4 
(64 KB splits, every document of every split read back and compared by `_id`):
   
   | Collection | Split key | Documents | Read |
   |---|---|---|---|
   | `_id` is 1000 ObjectId, 1000 string, 1000 long | `_id` (default) | 3000 | 
2520, 2340 and 2428 in three runs: 480, 660 and 572 lost |
   | `code` is a number, missing (500), null (500) or NaN (5) | `code`, indexed 
| 3000 | 2927 and 2849: 73 and 151 lost (NaN and numbers); 8995 in a third run: 
5995 read again |
   | `code` is a number, missing (500) or null (500) | `code` | 3000 | 6000: 
one split `{}` read every document again |
   | int `_id`, `match.query = "{_id: {$gte: 1500}}"` | `_id` | 1500 | 3000: 
1500 outside the match query |
   | `a.b` is a number | `a.b` | 3000 | 36000: all 12 splits `{}` |
   | `_id` is ObjectId only | `_id` | 20000 | 20000 |
   
   The samples are random, so the exact counts change from run to run. No error 
or warning is logged.
   
   ### SeaTunnel Version
   
   dev (`f1ab36f042`). The split code is unchanged since #9895.
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 2
     job.mode = "BATCH"
   }
   source {
     MongoDB {
       uri = "mongodb://127.0.0.1:27017/probe"
       database = "probe"
       collection = "mixed_id"
       schema = {
         fields {
           pad = string
         }
       }
     }
   }
   sink {
     Console {}
   }
   ```
   
   with a collection larger than `partition.split-size` whose `_id` values are 
ObjectIds, strings and numbers.
   
   ### Running Command
   
   ```shell
   SamplingSplitStrategy.split() against MongoDB 8.0.4, then 
find(split.getQuery()) for every split
   ```
   
   ### Error Exception
   
   ```log
   None. The job succeeds and reads fewer or more documents than the collection 
holds.
   ```
   
   ### Zeta or Flink or Spark Version
   
   All engines: the splits are created by the source's enumerator.
   
   ### Java or Scala Version
   
   JDK 17 locally.
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to