I suspect it is probably because the incoming rows when I joined with static frame can lead to variable degree of skewness over time and if so it is probably better to employ different join strategies at run time. But if you know your Dataset I believe you can just do broadcast join for your case!
Its been a while since I used spark so you might want to wait for more authoritative response Sent from my iPhone > On Jul 17, 2022, at 5:38 PM, Koert Kuipers <ko...@tresata.com> wrote: > > > i was surprised to find out that if a streaming dataframe is joined with a > static dataframe, that the static dataframe is re-shuffled for every > microbatch, which adds considerable overhead. > > wouldn't it make more sense to re-use the shuffle files? > > or if that is not possible then load the static dataframe into the > statestore? this would turn the join into a lookup (in rocksdb)? > > > CONFIDENTIALITY NOTICE: This electronic communication and any files > transmitted with it are confidential, privileged and intended solely for the > use of the individual or entity to whom they are addressed. If you are not > the intended recipient, you are hereby notified that any disclosure, copying, > distribution (electronic or otherwise) or forwarding of, or the taking of any > action in reliance on the contents of this transmission is strictly > prohibited. Please notify the sender immediately by e-mail if you have > received this email by mistake and delete this email from your system. > > Is it necessary to print this email? If you care about the environment like > we do, please refrain from printing emails. It helps to keep the environment > forested and litter-free.
