Guozhang Wang created KAFKA-6036:
------------------------------------
Summary: Enable logical materialization to physical materialization
Key: KAFKA-6036
URL: https://issues.apache.org/jira/browse/KAFKA-6036
Project: Kafka
Issue Type: Sub-task
Reporter: Guozhang Wang
Today whenever users specify a queryable store name for KTable, we would always
add a physical state store in the translated processor topology.
For some scenarios, we should consider not physically materialize the KTable
but only "logically" materialize it when you have some simple transformation
operations or even join operations that generated new KTables, and which needs
to be materialized with a state store, you can use the changelog topic of the
previous KTable and applies the transformation logic upon restoration instead
of creating a new changelog topic. For example:
{code}
table1 = builder.table("topic1");
table2 = table1.filter(..).join(table3); // table2 needs to be materialized for
joining
{code}
We can actually set the {{getter}} function of table2's materialized store, say
{{state2}} to be reading from {{topic1}} and then apply the filter operator,
instead of creating a new {{state2-changelog}} topic in this case.
We can come up with a general internal impl optimizations to determine when to
logically materialize, and whether we should actually allow users of DSL to
"hint" whether to materialize or not (it then may need a KIP).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
