JingsongLi commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1697842340
##########
paimon-core/src/main/java/org/apache/paimon/table/FileStoreTableFactory.java:
##########
@@ -89,6 +90,24 @@ public static FileStoreTable create(
fileIO, tablePath, tableSchema,
catalogEnvironment)
: new PrimaryKeyFileStoreTable(
fileIO, tablePath, tableSchema,
catalogEnvironment);
- return table.copy(dynamicOptions.toMap());
+ table = table.copy(dynamicOptions.toMap());
+
+ Options options = new Options(table.options());
+ String fallbackBranch = options.get(CoreOptions.SCAN_FALLBACK_BRANCH);
+ if (!StringUtils.isNullOrWhitespaceOnly(fallbackBranch)) {
+ Options branchOptions = new Options();
+ branchOptions.set(CoreOptions.BRANCH, fallbackBranch);
+ branchOptions.set(CoreOptions.SCAN_FALLBACK_BRANCH, "");
Review Comment:
You can introduce private method to `createNoWrappedTable`.
##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,90 @@ Run the following command:
{{< /tab >}}
{{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does
not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will
only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table
partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so
that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected
records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the
results of batch job.
+But if a partition (for example, today's partition) does not exist in its
result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set
`scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+ dt STRING NOT NULL,
+ name STRING NOT NULL,
+ amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+CALL sys.create_branch('default.T', 'test');
+
+-- set primary key and bucket number for the branch
+ALTER TABLE `T$branch_test` SET (
+ 'primary-key' = 'dt,name',
+ 'bucket' = '2'
+);
+
+-- set fallback branch
+ALTER TABLE T SET (
+ 'scan.fallback-branch' = 'test'
+);
+
+-- set changelog producer for the streaming branch, in case a streaming job
would like to read from it in the future
+ALTER TABLE `T$branch_test` SET (
Review Comment:
put two alter together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
