Hi Ryan,
My goal with this email thread is to discuss with the community if there
are better ideas (as I was told many other people tried to address this).
I'd consider this as a brainstorming email thread. Once we have a good
proposal, then we can go ahead with a SPIP.
Thanks,
Marco
Il giorno mer 12 dic 2018 alle ore 19:13 Ryan Blue <rb...@netflix.com> ha
scritto:
> Marco,
>
> I'm actually asking for a design doc that clearly states the problem and
> proposes a solution. This is a substantial change and probably should be an
> SPIP.
>
> I think that would be more likely to generate discussion than referring to
> PRs or a quick paragraph on the dev list, because the only people that are
> looking at it now are the ones already familiar with the problem.
>
> rb
>
> On Wed, Dec 12, 2018 at 2:05 AM Marco Gaido <marcogaid...@gmail.com>
> wrote:
>
>> Thank you all for your answers.
>>
>> @Ryan Blue <rb...@netflix.com> sure, let me state the problem more
>> clearly: imagine you have 2 dataframes with a common lineage (for instance
>> one is derived from the other by some filtering or anything you prefer).
>> And imagine you want to join these 2 dataframes. Currently, there is a fix
>> by Reynold which deduplicates the join condition in case the condition is
>> an equality one (please notice that in this case, it doesn't matter which
>> one is on the left and which one on the right). But if the condition
>> involves other comparisons, such as a ">" or a "<", this would result in an
>> analysis error, because the attributes on both sides are the same (eg. you
>> have the same id#3 attribute on both sides), and you cannot deduplicate
>> them blindly as which one is on a specific side matters.
>>
>> @Reynold Xin <r...@databricks.com> my proposal was to add a dataset id
>> in the metadata of each attribute, so that in this case we can distinguish
>> from which dataframe the attribute is coming from, ie. having the
>> DataFrames `df1` and `df2` where `df2` is derived from `df1`,
>> `df1.join(df2, df1("a") > df2("a"))` could be resolved because we would
>> know that the first attribute is taken from `df1` and so it has to be
>> resolved using it and the same for the other. But I am open to any approach
>> to this problem, if other people have better ideas/suggestions.
>>
>> Thanks,
>> Marco
>>
>> Il giorno mar 11 dic 2018 alle ore 18:31 Jörn Franke <
>> jornfra...@gmail.com> ha scritto:
>>
>>> I don’t know your exact underlying business problem, but maybe a graph
>>> solution, such as Spark Graphx meets better your requirements. Usually
>>> self-joins are done to address some kind of graph problem (even if you
>>> would not describe it as such) and is for these kind of problems much more
>>> efficient.
>>>
>>> Am 11.12.2018 um 12:44 schrieb Marco Gaido <marcogaid...@gmail.com>:
>>>
>>> Hi all,
>>>
>>> I'd like to bring to the attention of a more people a problem which has
>>> been there for long, ie, self joins. Currently, we have many troubles with
>>> them. This has been reported several times to the community and seems to
>>> affect many people, but as of now no solution has been accepted for it.
>>>
>>> I created a PR some time ago in order to address the problem (
>>> https://github.com/apache/spark/pull/21449), but Wenchen mentioned he
>>> tried to fix this problem too but so far no attempt was successful because
>>> there is no clear semantic (
>>> https://github.com/apache/spark/pull/21449#issuecomment-393554552).
>>>
>>> So I'd like to propose to discuss here which is the best approach for
>>> tackling this issue, which I think would be great to fix for 3.0.0, so if
>>> we decide to introduce breaking changes in the design, we can do that.
>>>
>>> Thoughts on this?
>>>
>>> Thanks,
>>> Marco
>>>
>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
