[
https://issues.apache.org/jira/browse/ARROW-10844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jorge Leitão updated ARROW-10844:
---------------------------------
Description:
The pseudo code
{code:java}
df = context.createDataFrame(...)
df1 = context.createDataFrame(...)
df.join(df1, ...)
{code}
currently does not work because we clone the {{ExecutionContextState}} from the
context to the `df`, causing the states from the left and right to share a
different state. In particular, `left` will not have the table registered on
the right, which means that its `collect` will fail.
We may need an Arc<Mutex<{{ExecutionContextState}}>> to share a common mutable
state across multiple DataFrames.
Note that the current example in `DataFrame` works because it shares the same
table. This won't happen if e.g. we use in-memory tables.
was:
The pseudo code
```
df = context.createDataFrame(...)
df1 = context.createDataFrame(...)
df.join(df1, ...)
```
currently does not work because we clone the {{ExecutionContextState}} from the
context to the `df`, causing the join statement to only have one of the
registered tables in the context state available (as it was not mutated by the
second `createdataFrame`).
We may need an Arc<Mutex<{{ExecutionContextState}}>> so that we have multiple
mutability of their attributes.
> [Rust] [DataFusion] join of two DataFrames is not possible
> ----------------------------------------------------------
>
> Key: ARROW-10844
> URL: https://issues.apache.org/jira/browse/ARROW-10844
> Project: Apache Arrow
> Issue Type: Bug
> Components: Rust, Rust - DataFusion
> Affects Versions: 3.0.0
> Reporter: Jorge Leitão
> Priority: Blocker
>
> The pseudo code
>
> {code:java}
> df = context.createDataFrame(...)
> df1 = context.createDataFrame(...)
> df.join(df1, ...)
> {code}
>
> currently does not work because we clone the {{ExecutionContextState}} from
> the context to the `df`, causing the states from the left and right to share
> a different state. In particular, `left` will not have the table registered
> on the right, which means that its `collect` will fail.
> We may need an Arc<Mutex<{{ExecutionContextState}}>> to share a common
> mutable state across multiple DataFrames.
> Note that the current example in `DataFrame` works because it shares the same
> table. This won't happen if e.g. we use in-memory tables.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
