ferenc-csaky commented on code in PR #84:
URL:
https://github.com/apache/flink-connector-mongodb/pull/84#discussion_r4616404083
##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java:
##########
@@ -186,8 +186,13 @@ public synchronized void close() throws Exception {
try {
doBulkWrite();
} catch (Exception e) {
- LOG.error("Writing records to MongoDB failed when closing
MongoWriter", e);
- throw new IOException("Writing records to MongoDB
failed.", e);
+ // Do not rethrow: delivery is guaranteed by flush(), so
records still
+ // buffered here are only written on a best-effort basis.
They are left
+ // over when the task failed or was cancelled, typically
the very batch
+ // whose flush just failed, and will be replayed from the
last checkpoint.
+ // Since FLINK-38180 (Flink 2.2) an exception thrown while
cleaning up a
+ // failed task is fatal and shuts down the whole
TaskManager.
+ LOG.warn("Writing records to MongoDB failed when closing
writer", e);
Review Comment:
The reasoning holds for at-least-once: every checkpoint and savepoint
flushes before its barrier, and end of input calls `flush(true)`, so anything
left here means the task failed or was cancelled. It doesn't hold for
`DeliveryGuarantee.NONE`. There's no flush on checkpoint, and
stop-with-savepoint without `--drain` never calls `endInput`, so this is the
only write of the buffered tail. Before, a failure here failed the stop. Now
the job finishes and those records are gone. NONE allows that, but the comment
should say so, and I'd log at ERROR since this line is the only trace of the
loss.
```suggestion
// Do not rethrow. With AT_LEAST_ONCE, flush() has
already written every
// record on each normal ending (end of input, or the
stop-with-savepoint
// barrier), so records still buffered here are left
over from a failed or
// cancelled task and are replayed from the last
checkpoint. With NONE, this
// is the only write of the buffered tail on
stop-with-savepoint without
// drain, and a failure here loses those records.
// Since Flink 2.2 (FLINK-38180) an exception thrown
while cleaning up a
// failed task is fatal and shuts down the whole
TaskManager.
LOG.error("Writing records to MongoDB failed when
closing writer", e);
```
##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableFactory.java:
##########
@@ -190,6 +194,8 @@ private static MongoWriteOptions
getWriteOptions(MongoConfiguration configuratio
return MongoWriteOptions.builder()
.setBatchSize(configuration.getBufferFlushMaxRows())
.setBatchIntervalMs(configuration.getBufferFlushIntervalMs())
+ .setOrderedWrites(configuration.isOrderedWrites())
Review Comment:
With `sink.ordered-writes=false` now reachable from SQL: on a table with a
primary key the sink emits upserting `UpdateOneModel`s and `DeleteOneModel`s,
and MongoDB doesn't guarantee the order of an unordered bulk write. A `-D(k)`
followed by `+I(k)` in the same batch can end with k deleted. Could you add a
sentence to the `sink.ordered-writes` rows in both the English and Chinese
docs, saying that with `false` a key changed more than once within one batch
can end up stale or deleted?
##########
flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableSink.java:
##########
@@ -109,6 +109,8 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context
context) {
.setBatchIntervalMs(writeOptions.getBatchIntervalMs())
.setDeliveryGuarantee(writeOptions.getDeliveryGuarantee())
.setMaxRetries(writeOptions.getMaxRetries())
+ .setOrderedWrites(writeOptions.isOrderedWrites())
Review Comment:
The same gap exists for `sink.retry.interval`. The factory puts it into
`writeOptions`, but `MongoSinkBuilder` has no setter for it, so the sink always
uses the 1 s default. It needs a new method on the `@PublicEvolving` builder,
so it doesn't belong in this PR. Could you open a follow-up JIRA, or fix it
here?
Passing the whole `MongoWriteOptions` to the builder instead of copying
fields one at a time would prevent the next one.
##########
flink-connector-mongodb/src/test/java/org/apache/flink/connector/mongodb/table/MongoDynamicTableSinkITCase.java:
##########
@@ -498,11 +519,108 @@ void testSinkWithCompositePrimaryKey() throws Exception {
assertThat(actual).isEqualTo(expected);
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSinkWithOrderedWrites(boolean orderedWrites) {
+ String database = "test";
+ String collection = "sink_with_ordered_writes_" + orderedWrites;
+
+ MongoCollection<Document> coll =
+ mongoClient.getDatabase(database).getCollection(collection);
+ coll.createIndex(Indexes.ascending("f1"), new
IndexOptions().unique(true));
+
+ TableEnvironment tEnv =
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+
+ Map<String, String> options = getConnectionOptions(uri, database,
collection);
+ options.put(MongoConnectorOptions.SINK_ORDERED_WRITES.key(),
String.valueOf(orderedWrites));
+
+ // Options that keep all rows of a test in one bulk write (no size or
interval based flush
+ // before the end of input) and fail the job on the first write error
instead of retrying.
Review Comment:
nit: the first row doesn't stay in the batch. `MongoWriter#lastSendTime`
starts at 0, so the interval check in `write()` is true for the first record
and `d1` is flushed on its own; `d1, d2` go out together at end of input. The
assertions still hold because the duplicate and `d2` share the second batch,
but reordering the rows to `d1, d2, d1` on the strength of this comment would
break the ordered case. Could you reword it here and at :591?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]