This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 4c6e75b2d2be Fix truncated return type in RFC-101 sample and typos
across docs (#19885)
4c6e75b2d2be is described below
commit 4c6e75b2d2bedc2e8ccee61d768a72a21bad292b
Author: Toyeshh Medikonda <[email protected]>
AuthorDate: Wed Sep 9 21:29:14 2026 -0500
Fix truncated return type in RFC-101 sample and typos across docs (#19885)
The HoodieRecordMerger snippet in RFC-101 declared fullOuterMerge as
returning `ist<Pair<HoodieRecord, Schema>>`; restore `List<...>` so the
sample compiles as written.
Also correct spelling in RFC design docs, the HFile format spec, the
Kafka Connect and integration-test READMEs, and the release guide
(e.g. PostgresSQL -> PostgreSQL, intial -> initial, requried -> required,
ture -> true, taskmanger -> taskmanager).
---
hudi-integ-test/README.md | 2 +-
hudi-io/hfile_format.md | 2 +-
hudi-kafka-connect/README.md | 2 +-
release/release_guide.md | 2 +-
rfc/rfc-101/rfc-101.md | 2 +-
rfc/rfc-107/rfc-107.md | 2 +-
rfc/rfc-27/rfc-27.md | 2 +-
rfc/rfc-38/rfc-38.md | 2 +-
rfc/rfc-39/rfc-39.md | 6 +++---
rfc/rfc-40/rfc-40.md | 4 ++--
rfc/rfc-42/rfc-42.md | 4 ++--
rfc/rfc-45/rfc-45.md | 4 ++--
rfc/rfc-51/rfc-51.md | 2 +-
rfc/rfc-56/rfc-56.md | 2 +-
rfc/rfc-65/rfc-65.md | 2 +-
rfc/rfc-68/rfc-68.md | 2 +-
rfc/rfc-76/rfc-76.md | 2 +-
rfc/rfc-78/rfc-78.md | 2 +-
rfc/rfc-80/rfc-80.md | 4 ++--
rfc/rfc-93/rfc-93.md | 2 +-
rfc/rfc-99/vector-appendix.md | 10 +++++-----
21 files changed, 31 insertions(+), 31 deletions(-)
diff --git a/hudi-integ-test/README.md b/hudi-integ-test/README.md
index d42f12cc2d26..7e7ee37416a6 100644
--- a/hudi-integ-test/README.md
+++ b/hudi-integ-test/README.md
@@ -491,7 +491,7 @@ simple-deltastreamer.yaml: simple 1 round dag for COW table.
simple-deltastreamer.yaml: simple 1 round dag for MOR table.
cow-clustering-example.yaml : dag with 3 rounds, in which inline clustering
will trigger during 2nd iteration.
cow-long-running-example.yaml : long running dag with 50 iterations. only 1
partition is used.
-cow-long-running-multi-partitions.yaml: long running dag wit 50 iterations
with multiple partitions.
+cow-long-running-multi-partitions.yaml: long running dag with 50 iterations
with multiple partitions.
```
To run test suite jobs for MOR table, pretty much any of these dags can be
used as is. Only change is with the
diff --git a/hudi-io/hfile_format.md b/hudi-io/hfile_format.md
index 3352fd39b7dd..e3340b741834 100644
--- a/hudi-io/hfile_format.md
+++ b/hudi-io/hfile_format.md
@@ -187,7 +187,7 @@ Each key-value pair has the following format:
Header:
- **Key Length**: 4 bytes, integer, length of the key part.
-- **Value Length**: 4 bytes, integer, lenghth of the value part.
+- **Value Length**: 4 bytes, integer, length of the value part.
Key:
diff --git a/hudi-kafka-connect/README.md b/hudi-kafka-connect/README.md
index f1811717f32e..e5ffee5ed4f9 100644
--- a/hudi-kafka-connect/README.md
+++ b/hudi-kafka-connect/README.md
@@ -137,7 +137,7 @@ bash setupKafka.sh -n <num_kafka_messages_per_batch> -b
<num_batches>
### 5 - Run the Sink connector worker (multiple workers can be run)
The Kafka connect is a distributed platform, with the ability to run one or
more workers (each running multiple tasks)
-that parallely process the records from the Kafka partitions for the same
topic. We provide a properties file with
+that parallelly process the records from the Kafka partitions for the same
topic. We provide a properties file with
default properties to start a Hudi connector.
Note that if multiple workers need to be run, the webserver needs to be
reconfigured for subsequent workers to ensure
diff --git a/release/release_guide.md b/release/release_guide.md
index 3175b63b96c6..c54163efdad1 100644
--- a/release/release_guide.md
+++ b/release/release_guide.md
@@ -635,7 +635,7 @@ Once the release candidate has been reviewed and approved
by the community, the
4. Repeat the steps from **_Generate Source Release (f) to Stage source
releases on [dist.apache.org](http://dist.apache.org/) (i)_**. Including
staging jars with the release version and uploading source release.
> **Note that make sure remove the -rc${RC_NUM} suffix when repeat the
above steps. and please also verify the steps. Ensure git tag is also done
without -rc${RC_NUM}**
5. One more step is to [deploy source code to
release](https://www.apache.org/legal/release-policy.html#upload-ci) dist.
[https://dist.apache.org/repos/dist/release/hudi](https://dist.apache.org/repos/dist/release/hudi).
- Only PMC will have access to this repo. So, if you are not a PMC, do get
help from somone who is.
+ Only PMC will have access to this repo. So, if you are not a PMC, do get
help from someone who is.
1. svn checkout https://dist.apache.org/repos/dist/release/hudi
--depth=immediates, if you would not checkout, please try svn checkout
https://dist.apache.org/repos/dist/release/hudi again.
2. Make a directory for the new release:
```shell
diff --git a/rfc/rfc-101/rfc-101.md b/rfc/rfc-101/rfc-101.md
index 6f30903dd324..e47cb19034d8 100644
--- a/rfc/rfc-101/rfc-101.md
+++ b/rfc/rfc-101/rfc-101.md
@@ -54,7 +54,7 @@ Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord
older, Schema oldSc
Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema,
HoodieRecord newer, Schema newSchema,
TypedProperties props) throws
IOException;
-ist<Pair<HoodieRecord, Schema>> fullOuterMerge(HoodieRecord older, Schema
oldSchema, HoodieRecord newer,
+List<Pair<HoodieRecord, Schema>> fullOuterMerge(HoodieRecord older, Schema
oldSchema, HoodieRecord newer,
Schema newSchema,
TypedProperties props) throws IOException;
boolean shouldFlush(HoodieRecord record, Schema schema, TypedProperties props)
throws IOException;
diff --git a/rfc/rfc-107/rfc-107.md b/rfc/rfc-107/rfc-107.md
index 56f5f0baa1a2..901bdfc735db 100644
--- a/rfc/rfc-107/rfc-107.md
+++ b/rfc/rfc-107/rfc-107.md
@@ -217,7 +217,7 @@ For partitioned RLI, the cache implements automatic
eviction of cold partitions:
- Each column family tracks the **last access timestamp** (last time a record
was written to or looked up in that partition).
- The TTL should be set based on the workload's partition access pattern. For
daily-partitioned event data, a TTL of 3–7 days is typical.
-- Given we need to consider the ongoing checkpoint, the cleanup process of
droping column families whose last access exceeds the configured TTL should
happen right after checkpoint completion.
+- Given we need to consider the ongoing checkpoint, the cleanup process of
dropping column families whose last access exceeds the configured TTL should
happen right after checkpoint completion.
Configuration:
diff --git a/rfc/rfc-27/rfc-27.md b/rfc/rfc-27/rfc-27.md
index 1454dce0d4b5..ec05ccfb9a90 100644
--- a/rfc/rfc-27/rfc-27.md
+++ b/rfc/rfc-27/rfc-27.md
@@ -60,7 +60,7 @@ RFC-15 added metadata table support to Hudi for optimized
file listing. RFC-37 i
as another partition to metadata table. This RFC will piggyback on the column
stats partition that RFC-37 will be adding
to metadata table.
-Notes: Effectiveness of the index will be proportional to how data is layed
out. If every file contains data for
+Notes: Effectiveness of the index will be proportional to how data is laid
out. If every file contains data for
commonly specified query predicate, index may not be very effective.
## Implementation
diff --git a/rfc/rfc-38/rfc-38.md b/rfc/rfc-38/rfc-38.md
index d007bd0b663d..feda07d94896 100644
--- a/rfc/rfc-38/rfc-38.md
+++ b/rfc/rfc-38/rfc-38.md
@@ -242,7 +242,7 @@ with StagingTableCatalog {
}
override def alterTable(Identifier ident, TableChange... changes): Table = {
- // check schema compability
+ // check schema compatibility
// HoodieDatasouceTable
}
diff --git a/rfc/rfc-39/rfc-39.md b/rfc/rfc-39/rfc-39.md
index 9dc6335b2b9a..5fab69fbcf2c 100644
--- a/rfc/rfc-39/rfc-39.md
+++ b/rfc/rfc-39/rfc-39.md
@@ -42,7 +42,7 @@ The first time it connects to a PostgreSQL server or cluster,
the connector take
-The schema of the events for debezium consists of a before, after, source, op
and ts\_ms. The `before` field contains the values of the row before the
operation took place. And `after` field contains the values of the original
database row after the operation took place. The operation is specified in `op`
field, which can be either `r` (initial snapshot), `c` (insert), `u` (update)
or `d` (delete). In case of insert, the `before` field will be null which for a
delete, the `after` field w [...]
+The schema of the events for debezium consists of a before, after, source, op
and ts\_ms. The `before` field contains the values of the row before the
operation took place. And `after` field contains the values of the original
database row after the operation took place. The operation is specified in `op`
field, which can be either `r` (initial snapshot), `c` (insert), `u` (update)
or `d` (delete). In case of insert, the `before` field will be null which for a
delete, the `after` field w [...]
@@ -65,7 +65,7 @@ To incrementally ingest the changelogs from the debezium
connector, we propose t
Since we change the schema of the incoming record in the source class, we have
to provide a schema for the target record. We propose to implement
DebeziumAvroSource.java as a RowSource and allow spark to infer the schema of
the transformed record. An alternative approach is to implement a
DebeziumSchemaRegistryProvider.java class that extends the current
SchemaRegistryProvider.java, and implements the method getTargetSchema . It
constructs the target schema from the original schema by in [...]
-To ensure proper de-dup, merging, and hard deletes of the records, we
implement a custom AvroPayload class for debeizum: `DebeziumAvroPayload.java.`
During writes, we check if the `op` field of the record is `d` , we return an
empty payload to ensure the record is deleted in storage. In the case of
`preCombine` or `combineAndGetUpdateValue` (merge handling of records), we
return the existing stored record if the `LSN` (in case of PostgresSQL) of the
existing record is higher than the new [...]
+To ensure proper de-dup, merging, and hard deletes of the records, we
implement a custom AvroPayload class for debeizum: `DebeziumAvroPayload.java.`
During writes, we check if the `op` field of the record is `d` , we return an
empty payload to ensure the record is deleted in storage. In the case of
`preCombine` or `combineAndGetUpdateValue` (merge handling of records), we
return the existing stored record if the `LSN` (in case of PostgreSQL) of the
existing record is higher than the newl [...]
###Handling merges with Postgres Toast Columns
@@ -102,4 +102,4 @@ This is a new feature specific to Debezium CDC use case,
and should not impact e
## Test Plan
-We plan to test the Debezium source by setting up a AWS RDS instance of
PostgresSQL, debezium connector using strimzi operator on k8s and a AWS MSK
kafka cluster. We will test for correctness by performing SQL based DDL
operations, such as insert, update and deletions on multiple records/ rows in
the Postgres DB, and query the hudi table to validate that the operations took
effect on the records in the hudi table.
\ No newline at end of file
+We plan to test the Debezium source by setting up a AWS RDS instance of
PostgreSQL, debezium connector using strimzi operator on k8s and a AWS MSK
kafka cluster. We will test for correctness by performing SQL based DDL
operations, such as insert, update and deletions on multiple records/ rows in
the Postgres DB, and query the hudi table to validate that the operations took
effect on the records in the hudi table.
\ No newline at end of file
diff --git a/rfc/rfc-40/rfc-40.md b/rfc/rfc-40/rfc-40.md
index 252507155126..11b5bd5960ac 100644
--- a/rfc/rfc-40/rfc-40.md
+++ b/rfc/rfc-40/rfc-40.md
@@ -113,7 +113,7 @@ public class HudiSplit
The split manager will partition the data for a table into the individual
chunks that Trino will distribute to workers
for processing. This is where the partition loader logic will reside. While
listing the files for each Hudi partition,
the split manager will create one or more splits per file. Additionally, split
generation is dynamic based on size to
-futher improve the performance (see [query planning
optimization](#query-planning-optimization) for more details).
+further improve the performance (see [query planning
optimization](#query-planning-optimization) for more details).
During query execution, the Trino coordinator tracks all splits available for
processing and the locations where tasks
are running on workers and processing splits. As tasks finish processing and
are producing more splits for downstream
@@ -272,7 +272,7 @@ Hudi, so we need to either use the Hudi APIs or write
custom Spark jobs to migra
- When will we remove the existing behavior?
We are not proposing to remove the existing behavior. We hope that we will
have a critical mass of users who will like
-to use the new Hudi connector. That said, we whould continue to support the
current integration.
+to use the new Hudi connector. That said, we would continue to support the
current integration.
## Test Plan
diff --git a/rfc/rfc-42/rfc-42.md b/rfc/rfc-42/rfc-42.md
index 97f4146f9a71..f8ac35722eed 100644
--- a/rfc/rfc-42/rfc-42.md
+++ b/rfc/rfc-42/rfc-42.md
@@ -59,7 +59,7 @@ This simple yet effective design reduce the time complexity
of the key lookup to
-However, there are also some limitions.
+However, there are also some limitations.
As described in
[RFC-29](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+29%3A+Hash+Index),
the one-one mapping between buckets and file groups may cause data skew and
doesn't scale well.
One solution to address these problems is allowing one bucket to have multiple
file groups, which in turn requires indexing to be performed inside each bucket.
@@ -131,7 +131,7 @@ Only three operations will modify the hashing metadata:
- Clean service: removes outdated metadata files in the same way as removing
old file groups.
Though storing hashing metadata in the `.hoodie` path is a straightforward
solution, it should be put into hudi's metadata table ultimately.
-And the clean service is no loger necessary.
+And the clean service is no longer necessary.
Old version hashing metadata will also be cleanup automatically since the
metadata table itself is a hudi table.
### Bucket Resizing (Splitting & Merging)
diff --git a/rfc/rfc-45/rfc-45.md b/rfc/rfc-45/rfc-45.md
index f79dd896a09e..87c965dffb15 100644
--- a/rfc/rfc-45/rfc-45.md
+++ b/rfc/rfc-45/rfc-45.md
@@ -154,7 +154,7 @@ Further, suppose there were two inflight writers Writer1
and Writer2 (with
inflight instants `t1` and `t2` respectively) while the indexing was requested
or inflight. In this case, the writers will check for pending index action and
find a pending instant `t3`. Now, if the metadata index creation is pending,
-which means indexer has already intialized a filegroup, then each writer will
+which means indexer has already initialized a filegroup, then each writer will
create log files in the same filegroup for the metadata index update. This will
happen within the existing data table lock.
@@ -358,7 +358,7 @@ the data table.
- If we are changing behavior how will we phase out the older behavior?
-The changes will be backward-compatible and if the async indexing is diabled
+The changes will be backward-compatible and if the async indexing is disabled
then the existing behavior of MDT creation and updates will be used.
- If we need special migration tools, describe them here.
diff --git a/rfc/rfc-51/rfc-51.md b/rfc/rfc-51/rfc-51.md
index 29115b463445..fe97dd33b336 100644
--- a/rfc/rfc-51/rfc-51.md
+++ b/rfc/rfc-51/rfc-51.md
@@ -89,7 +89,7 @@ To perform CDC queries, users need to set
`hoodie.datasource.query.incremental.f
|--------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------|
| hoodie.datasource.query.type | `snapshot` | set to
`incremental` for incremental query.
|
| hoodie.datasource.query.incremental.format | `latest_state` | `latest_state`
(current incremental query behavior) returns the latest records' values. Set to
`cdc` to return the full CDC results. |
-| hoodie.datasource.read.begin.instanttime | - | requried.
|
+| hoodie.datasource.read.begin.instanttime | - | required.
|
| hoodie.datasource.read.end.instanttime | - | optional.
|
## When `supplemental.logging.mode=KEY_OP`
diff --git a/rfc/rfc-56/rfc-56.md b/rfc/rfc-56/rfc-56.md
index ed400072d488..241779a57390 100644
--- a/rfc/rfc-56/rfc-56.md
+++ b/rfc/rfc-56/rfc-56.md
@@ -261,7 +261,7 @@ This RFC adds a feature flag and three new configs to
control the behavior of ea
for `TIMELINE_SERVER_BASED` markers. The class name of the early conflict
detection strategy to use. This should be a
subclass of
`org.apache.hudi.common.conflict.detection.EarlyConflictDetectionStrategy`.
3. `hoodie.write.concurrency.early.conflict.check.commit` default: `false`.
Set true if users are sensitive to conflict
- detection. When set to ture, Hudi is able to check commit conflict during
early conflict detection.
+ detection. When set to true, Hudi is able to check commit conflict during
early conflict detection.
4. `hoodie.write.concurrency.async.conflict.detector.initial_delay_ms`
default: `0L`. Used for timeline-server-based
markers with `AsyncTimelineServerBasedDetectionStrategy`. The time in
milliseconds to delay the first execution of
async marker-based conflict detection.
diff --git a/rfc/rfc-65/rfc-65.md b/rfc/rfc-65/rfc-65.md
index b04b82c48dd3..fec8f4e5db96 100644
--- a/rfc/rfc-65/rfc-65.md
+++ b/rfc/rfc-65/rfc-65.md
@@ -105,7 +105,7 @@ For 1.0.0 and later hudi version which supports efficient
completion time querie
### Apply different strategies for different partitions
-For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions under `product=1` they want to keep for
30 days while for partitions under `product=2` they want to keep for 7 days
only.
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fields(productId, day). For partitions under `product=1` they want to keep for
30 days while for partitions under `product=2` they want to keep for 7 days
only.
For the first version of TTL management, we do not plan to implement a
complicated strategy (For example, use an array to store strategies, introduce
partition regex etc.). Instead, we add a new abstract method
`getPartitionPathsForTTL` in `PartitionTTLStrategy` and provides a new config
`hoodie.partition.ttl.partition.selected`.
diff --git a/rfc/rfc-68/rfc-68.md b/rfc/rfc-68/rfc-68.md
index 598ff09f4c11..f9ecd42ed762 100644
--- a/rfc/rfc-68/rfc-68.md
+++ b/rfc/rfc-68/rfc-68.md
@@ -49,7 +49,7 @@ a. Add one more member variable(Integer rowGroupId) into the
class HoodieRecordL
protected Integer rowGroupNum;
}</pre>
-b. Number of rowgroup of a Parquet starts from 0 which continously increases
util BlockSize reaches `hoodie.parquet.block.size`. Since every record in
parquet belongs to a rowgroup, we can simply use parquet API to locate rowgroup
num of new record which needs to be written into corresponding parquet file,
and then record rowgroup num into hoodieRecordLocation of each hoodieRecord.
HoodieRecordLocations will be collected into WriteStatus which will be updated
to the index on batch.
+b. Number of rowgroup of a Parquet starts from 0 which continuously increases
until BlockSize reaches `hoodie.parquet.block.size`. Since every record in
parquet belongs to a rowgroup, we can simply use parquet API to locate rowgroup
num of new record which needs to be written into corresponding parquet file,
and then record rowgroup num into hoodieRecordLocation of each hoodieRecord.
HoodieRecordLocations will be collected into WriteStatus which will be updated
to the index on batch.
c. At phase of tagging index, rowgroup num will be queried out, so that they
can be used to accelerate updating files.
diff --git a/rfc/rfc-76/rfc-76.md b/rfc/rfc-76/rfc-76.md
index e9f176f1d5f7..699cac24d8de 100644
--- a/rfc/rfc-76/rfc-76.md
+++ b/rfc/rfc-76/rfc-76.md
@@ -109,7 +109,7 @@ Based on our experiments, here is the storage comparison
across different key en
#### Analysis
Both uncompressed and compressed sizes of record key columns in UUID6/7 are
much bigger than our original formats, which means we can discard them.
-Compared with the base line format Original, Base64 and ASCII formats can
produce better results based on the storage usage.Specifially, Base64 format
can produce around 17% of storage reduction after Parquet compression, and
ASCII can produce around 28% of reduction. However, to extract relevant bytes
and do the bit distribution and encoding, Base64 and ASCII can definitely
require more CPU powers during writings (400x).
+Compared with the base line format Original, Base64 and ASCII formats can
produce better results based on the storage usage. Specifically, Base64 format
can produce around 17% of storage reduction after Parquet compression, and
ASCII can produce around 28% of reduction. However, to extract relevant bytes
and do the bit distribution and encoding, Base64 and ASCII can definitely
require more CPU powers during writings (400x).
#### Consensus
So considering the storage size and runtimes across different encoding formats
we will settle with the original format ie. "${commit_timestamp}_${spark
partition id}, ${row Id}" for our auto record key generation.
diff --git a/rfc/rfc-78/rfc-78.md b/rfc/rfc-78/rfc-78.md
index 186407abe30b..251a1d3a0f85 100644
--- a/rfc/rfc-78/rfc-78.md
+++ b/rfc/rfc-78/rfc-78.md
@@ -73,7 +73,7 @@ Hence, the migration protocol is key to ensure a smooth
migration experience for
- 1.x writer should be able to write in both the table versions tables (table
version 6 and the current version) w/o any
loss in 0.x functionality and guaranteeing no data inconsistencies.
But for new features that was introduced in 1.x, we may not be able to
support all of them.
-- In this case, we explicitly request users to not turn on these features
untill all readers are completely migrated to
+- In this case, we explicitly request users to not turn on these features
until all readers are completely migrated to
1.x so as to not break any readers as applicable.
Connecting back to our example above, lets see how the migration might look
like for an existing user.
diff --git a/rfc/rfc-80/rfc-80.md b/rfc/rfc-80/rfc-80.md
index 9ec39f36ee7b..b0af42700237 100644
--- a/rfc/rfc-80/rfc-80.md
+++ b/rfc/rfc-80/rfc-80.md
@@ -145,8 +145,8 @@ The entire reading process involves a large amount of data
merging, but because
#### Engine reads pseudo process
-1) The engine itself delivers the data files that need to be scanned to
executor/woker/taskmanger.
-2) executor/worker/taskmanger calls Hudi’s rowReader interface and passes in
column clipping and filter conditions to rowReader.
+1) The engine itself delivers the data files that need to be scanned to
executor/worker/taskmanager.
+2) executor/worker/taskmanager calls Hudi’s rowReader interface and passes in
column clipping and filter conditions to rowReader.
3) The Hudi kernel completes the data reading of rowReader and returns
complete data. The data format is Avro.
4) The engine gets the Avro format data and needs to convert it into the data
format it needs. For example, spark needs to be converted into unsaferow, hetu
into block, flink into row, and hive into arrayWritable.
diff --git a/rfc/rfc-93/rfc-93.md b/rfc/rfc-93/rfc-93.md
index 3a5f0aef540b..7bae81a103c5 100644
--- a/rfc/rfc-93/rfc-93.md
+++ b/rfc/rfc-93/rfc-93.md
@@ -116,7 +116,7 @@ The key points here are:
1. Hudi Writers continue to use the native file system management. This means
the way Hudi organizes files as file-groups and file-slices and Hudi's index
for writing will continue to be the same.
2. Hudi will continue to record actions in timeline (.hoodie/timeline).
3. The external table format will be configured as part of table properties to
ensure all writers can recognize it.
-4. When each of Hudi's actions (write, clean, rollback,..) complete, Hudi will
call ther external table format plugin. It is the plugin's responsibility to
record necessary metadata for external format readers to query this table.
+4. When each of Hudi's actions (write, clean, rollback,..) complete, Hudi will
call the external table format plugin. It is the plugin's responsibility to
record necessary metadata for external format readers to query this table.
5. Hudi will use the external table format plugin's timeline implementation
which uses the state stored in its metadata as the source of truth. For
example, an iceberg plugin needs to create a manifest file when action
completes and provides a timeline that honors the state in its snapshot files.
Hudi uses this timeline to determine what actions completed and what failed
(for rollback).
6. The external plugin also needs to provide LockProvider and Conflict
Resolution implementations that allows the concurrency controls supported by
the external table format to be exercised (for example: optimistic concurrency
control).
7. The plugin also provides capability to allow its metadata to be used by the
Hudi writer.
diff --git a/rfc/rfc-99/vector-appendix.md b/rfc/rfc-99/vector-appendix.md
index e8bff5479653..de7948b5b885 100644
--- a/rfc/rfc-99/vector-appendix.md
+++ b/rfc/rfc-99/vector-appendix.md
@@ -22,7 +22,7 @@ This section captures additional research and design notes
for supporting a VECT
### Initial scope
-The intial use case we are targeting for `VECTOR` within Hudi,
+The initial use case we are targeting for `VECTOR` within Hudi,
is to enable KNN style vector search functionality to be performed on
blobs(large text, images, audio, video) alongside their generated vector
embeddings.
Typically vector search is popular for Retrieval-Augmented Generation (RAG)
applications
which provide relevant context to an LLM in order to improve its accuracy when
answering user queries.
@@ -41,9 +41,9 @@ The vector embeddings generated by frontier models are
usually in the form of an
* [(3, 0.44), (107, 1.2), (9012, -0.7)]
* The “dimension” is still D, but the stored length is nnz (number of
non-zeros), typically nnz << D.
-Sparse vectors become important for other types of hybrid/lexical-style
retrieval which is not targeted for the intial scope,
-as that requires running different algorithms such as (TF-IDF or BM25) which
is different from the intial use case of KNN style search.
-Hence this RFC has seperated both into two distinct types one for VECTOR
(dense) and one for SPARSE_VECTOR, we will for now spend time on VECTOR dense
case.
+Sparse vectors become important for other types of hybrid/lexical-style
retrieval which is not targeted for the initial scope,
+as that requires running different algorithms such as (TF-IDF or BM25) which
is different from the initial use case of KNN style search.
+Hence this RFC has separated both into two distinct types one for VECTOR
(dense) and one for SPARSE_VECTOR, we will for now spend time on VECTOR dense
case.
### Vector Schema constraints
@@ -52,7 +52,7 @@ Hence this RFC has seperated both into two distinct types one
for VECTOR (dense)
- All values within the VECTOR column must have the same **dimension** i.e
(number of elements within the vector), as this is needed to perform
cosine/L2/dot-product correctly.
- There should be no null elements within the vector at write time.
- VECTOR must have an "element type" which can be one of `FLOAT`, `DOUBLE` or
`INT8`.
-- We also want to keep a property around such as `storageBacking` which lets
the writers know how to serialize the vector to disk. For an intial approach we
will start with a fixed bytes approach covered below.
+- We also want to keep a property around such as `storageBacking` which lets
the writers know how to serialize the vector to disk. For an initial approach
we will start with a fixed bytes approach covered below.
See the following avro schema model as a general example:
```
