ZZZxDong commented on PR #8837: URL: https://github.com/apache/paimon/pull/8837#issuecomment-5140317925
Thanks, all three are valid — fixed. - Quoted the column names before passing them to `col`, so a name like `` `a.b` `` is no longer parsed as a nested field. - The literal path no longer materializes the cartesian product on the driver: it builds one single-column DataFrame per pk column and cross joins them, so only the literals that the condition itself carries stay in the driver (they are part of the plan anyway) and Spark computes the product. The intermediate `Seq[Map]` is gone. - The row-count limit is now checked with division instead of multiplying first, so it can not overflow, and the default is lowered from 1,000,000 to 10,000 — deleting a large set of keys should go through `pk IN (subquery)`, which never touches the driver. On "why cache in memory": for a literal condition the values are already in the driver as part of the plan, so what was removed is the extra product materialization rather than the literals themselves. Nothing is cached for the subquery path. Side note found while testing this: `PaimonSparkWriter` computes the bucket with an unquoted `col(...)` too, so a table whose bucket key contains a dot already fails on INSERT. That is pre-existing and out of scope here — happy to send a separate fix if you want. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
