ferenc-csaky commented on code in PR #84:
URL: 
https://github.com/apache/flink-connector-mongodb/pull/84#discussion_r4616404083


##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java:
##########
@@ -186,8 +186,13 @@ public synchronized void close() throws Exception {
                 try {
                     doBulkWrite();
                 } catch (Exception e) {
-                    LOG.error("Writing records to MongoDB failed when closing 
MongoWriter", e);
-                    throw new IOException("Writing records to MongoDB 
failed.", e);
+                    // Do not rethrow: delivery is guaranteed by flush(), so 
records still
+                    // buffered here are only written on a best-effort basis. 
They are left
+                    // over when the task failed or was cancelled, typically 
the very batch
+                    // whose flush just failed, and will be replayed from the 
last checkpoint.
+                    // Since FLINK-38180 (Flink 2.2) an exception thrown while 
cleaning up a
+                    // failed task is fatal and shuts down the whole 
TaskManager.
+                    LOG.warn("Writing records to MongoDB failed when closing 
writer", e);

Review Comment:
   The reasoning holds for at-least-once: every checkpoint and savepoint 
flushes before its barrier, and end of input calls `flush(true)`, so anything 
left here means the task failed or was cancelled. It doesn't hold for 
`DeliveryGuarantee.NONE`. There's no flush on checkpoint, and 
stop-with-savepoint without `--drain` never calls `endInput`, so this is the 
only write of the buffered tail. Before, a failure here failed the stop. Now 
the job finishes and those records are gone. NONE allows that, but the comment 
should say so, and I'd log at ERROR since this line is the only trace of the 
loss.
   
   ```suggestion
                       // Do not rethrow. With AT_LEAST_ONCE, flush() has 
already written every
                       // record on each normal ending (end of input, or the 
stop-with-savepoint
                       // barrier), so records still buffered here are left 
over from a failed or
                       // cancelled task and are replayed from the last 
checkpoint. With NONE, this
                       // is the only write of the buffered tail on 
stop-with-savepoint without
                       // drain, and a failure here loses those records.
                       // Since Flink 2.2 (FLINK-38180) an exception thrown 
while cleaning up a
                       // failed task is fatal and shuts down the whole 
TaskManager.
                       LOG.error("Writing records to MongoDB failed when 
closing writer", e);
   ```



##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableFactory.java:
##########
@@ -190,6 +194,8 @@ private static MongoWriteOptions 
getWriteOptions(MongoConfiguration configuratio
         return MongoWriteOptions.builder()
                 .setBatchSize(configuration.getBufferFlushMaxRows())
                 .setBatchIntervalMs(configuration.getBufferFlushIntervalMs())
+                .setOrderedWrites(configuration.isOrderedWrites())

Review Comment:
   With `sink.ordered-writes=false` now reachable from SQL: on a table with a 
primary key the sink emits upserting `UpdateOneModel`s and `DeleteOneModel`s, 
and MongoDB doesn't guarantee the order of an unordered bulk write. A `-D(k)` 
followed by `+I(k)` in the same batch can end with k deleted. Could you add a 
sentence to the `sink.ordered-writes` rows in both the English and Chinese 
docs, saying that with `false` a key changed more than once within one batch 
can end up stale or deleted?



##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableSink.java:
##########
@@ -109,6 +109,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context 
context) {
                         .setBatchIntervalMs(writeOptions.getBatchIntervalMs())
                         
.setDeliveryGuarantee(writeOptions.getDeliveryGuarantee())
                         .setMaxRetries(writeOptions.getMaxRetries())
+                        .setOrderedWrites(writeOptions.isOrderedWrites())

Review Comment:
   The same gap exists for `sink.retry.interval`. The factory puts it into 
`writeOptions`, but `MongoSinkBuilder` has no setter for it, so the sink always 
uses the 1 s default. It needs a new method on the `@PublicEvolving` builder, 
so it doesn't belong in this PR. Could you open a follow-up JIRA, or fix it 
here?
   
   Passing the whole `MongoWriteOptions` to the builder instead of copying 
fields one at a time would prevent the next one.



##########
flink-connector-mongodb/src/test/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableSinkITCase.java:
##########
@@ -498,11 +519,108 @@ void testSinkWithCompositePrimaryKey() throws Exception {
         assertThat(actual).isEqualTo(expected);
     }
 
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})
+    void testSinkWithOrderedWrites(boolean orderedWrites) {
+        String database = "test";
+        String collection = "sink_with_ordered_writes_" + orderedWrites;
+
+        MongoCollection<Document> coll =
+                mongoClient.getDatabase(database).getCollection(collection);
+        coll.createIndex(Indexes.ascending("f1"), new 
IndexOptions().unique(true));
+
+        TableEnvironment tEnv = 
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+
+        Map<String, String> options = getConnectionOptions(uri, database, 
collection);
+        options.put(MongoConnectorOptions.SINK_ORDERED_WRITES.key(), 
String.valueOf(orderedWrites));
+
+        // Options that keep all rows of a test in one bulk write (no size or 
interval based flush
+        // before the end of input) and fail the job on the first write error 
instead of retrying.

Review Comment:
   nit: the first row doesn't stay in the batch. `MongoWriter#lastSendTime` 
starts at 0, so the interval check in `write()` is true for the first record 
and `d1` is flushed on its own; `d1, d2` go out together at end of input. The 
assertions still hold because the duplicate and `d2` share the second batch, 
but reordering the rows to `d1, d2, d1` on the strength of this comment would 
break the ordered case. Could you reword it here and at :591?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to