[
https://issues.apache.org/jira/browse/KAFKA-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Bausov updated KAFKA-7419:
------------------------------------
Comment: was deleted
(was: Hi, John!
Thanks for fast reply. Consider we have stream of exchange trades (20 pre
second):
{code:java|title=TradeMessage.java}
public class TradeMessage {
private long timestamp;
private BigDecimal price;
private BigDecimal quantity;
}{code}
Than each record should be converted to Ticker stream:
{code:java|title=Ticker.java}
public class Ticker {
private BigDecimal open;
private BigDecimal last;
private BigDecimal low;
private BigDecimal high;
private BigDecimal volume24; // 24h trades quantities sum
}{code}
If we use
{code:java}
.windowedBy(TimeWindows.of(24 * 60 * 60 * 1000).advanceBy(1000)) // sliding 24h
window with 1s step{code}
we get 86400 windowed frames. Too much resources and slow performance for such
operation as rolling sum (volume24).
Think is possible to use more simple algorithm: count total volume from the
beginning (reduce or aggregate each coming record), than volume24 =
totalVolumeNow - totalVolumeOfRecord24hAgo. One request to StateStore. Going to
realise this algorithm via Processor API. Maybe you can advice something more
efficient?
P.S. Also may be you have thoughts how to count rolling min and max, also for
sliding window with 1s step:
{code:java|title=Ticker.java}
public class Ticker {
private BigDecimal open;
private BigDecimal last;
private BigDecimal low; // min trade price for last 24h
private BigDecimal high; // max trade price for last 24h
private BigDecimal volume24;
}{code})
> Rolling sum for high frequency sream
> ------------------------------------
>
> Key: KAFKA-7419
> URL: https://issues.apache.org/jira/browse/KAFKA-7419
> Project: Kafka
> Issue Type: Wish
> Components: streams
> Reporter: Stanislav Bausov
> Priority: Minor
>
> Have a task to count 24h market volume for high frequency trades stream. And
> there is no solution out of the box. Windowing is not an option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
