jihoonson commented on a change in pull request #10748:
URL: https://github.com/apache/druid/pull/10748#discussion_r570681348
##########
File path: docs/ingestion/native-batch.md
##########
@@ -192,7 +192,8 @@ that range if there's some stray data with unexpected
timestamps.
|--------|-----------|-------|---------|
|type|The task type, this should always be `index_parallel`.|none|yes|
|inputFormat|[`inputFormat`](./data-formats.md#input-format) to specify how to
parse input data.|none|yes|
-|appendToExisting|Creates segments as additional shards of the latest version,
effectively appending to the segment set instead of replacing it. The current
limitation is that you can append to any datasources regardless of their
original partitioning scheme, but the appended segments should be partitioned
using the `dynamic` partitionsSpec.|false|no|
+|appendToExisting|Creates segments as additional shards of the latest version,
effectively appending to the segment set instead of replacing it. This means
that you can append new segments to any datasource regardless of its original
partitioning scheme.
+`appendToExisting` is incompatible with `forceGuaranteedRollup` and its
associated partitioning types, for example `hashed`. Therefore, you must use
the `dynamic` partitioning type for the appended segments. Otherwise the task
fails with the following error: "forceGuaranteedRollup must be set".|false|no|
Review comment:
Actually there are a couple cases when something can go wrong.
- `appendToExisting`: true, `forceGuaranteedRollup`: false, `partitioning` =
`dynamic`: this is the only valid setup.
- `appendToExisting`: true, `forceGuaranteedRollup`: false, `partitioning` =
`hashed`: error with a message of `DynamicPartitionsSpec must be used for
best-effort rollup`.
- `appendToExisting`: true, `forceGuaranteedRollup`: true, `partitioning` =
`dynamic`: error with a message of `DynamicPartitionsSpec cannot be used for
perfect rollup`.
- `appendToExisting`: true, `forceGuaranteedRollup`: true, `partitioning` =
`hashed`/`single_dim`: error with a message of `Perfect rollup cannot be
guaranteed when appending to existing dataSources`.
I'm not sure it's reasonable to list out all these errors here. It's hard to
improve the error message for now because we don't know exactly what users want
to do. Maybe it's better to simply say the task will fail.
BTW, `forceGuaranteedRollup` seems not very useful so we can get rid of it
in the future. Once we remove it, then the error message can be simplified to
something like `appendToExisting cannot be set with hashed partitioning`.
##########
File path: docs/ingestion/native-batch.md
##########
@@ -192,7 +192,8 @@ that range if there's some stray data with unexpected
timestamps.
|--------|-----------|-------|---------|
|type|The task type, this should always be `index_parallel`.|none|yes|
|inputFormat|[`inputFormat`](./data-formats.md#input-format) to specify how to
parse input data.|none|yes|
-|appendToExisting|Creates segments as additional shards of the latest version,
effectively appending to the segment set instead of replacing it. The current
limitation is that you can append to any datasources regardless of their
original partitioning scheme, but the appended segments should be partitioned
using the `dynamic` partitionsSpec.|false|no|
+|appendToExisting|Creates segments as additional shards of the latest version,
effectively appending to the segment set instead of replacing it. This means
that you can append new segments to any datasource regardless of its original
partitioning scheme.
Review comment:
Did you intend to break the line here? This actually breaks the table
(https://github.com/apache/druid/blob/9331331b857419c6ca45c15a1852058355019c87/docs/ingestion/native-batch.md#ioconfig).
You need to add `<br/>` to break the line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
