Re: Handling Large Broadcast States

2021-06-18 Thread Timo Walther
Hi Rion, as far as I know we also don't support broadcast streaming joins in Table API/SQL. Are you sure that you need a broadcast pattern? Or would a regular hash join using connect() with a CoProcessFunction also work for you? Maybe with an artifical key to spread the load more evently?

Re: Handling Large Broadcast States

2021-06-18 Thread Piotr Nowojski
Hi, As far as I know there are no plans to support other state backends with BroadcastState. I don't know about any particular technical limitation, it probably just hasn't been done. Also I don't know how much effort that would be. Probably it wouldn't be easy. Timo, can you chip in how for

Handling Large Broadcast States

2021-06-16 Thread Rion Williams
Hey Flink folks, I was discussing the use of the Broadcast Pattern with some colleagues today for a potential enrichment use-case and noticed that it wasn’t currently backed by RocksDB. This seems to indicate that it would be solely limited to the memory allocated, which might not support a