Sure. I will send prototypical query tomorrow. Though its difficult to simulate issue using unit test , but I think the issue is Rdd.isIndeterminate is not returning true for the query. As a result, on retry, the shuffle stage is not reattempted fully. And rdd is not returning inDeterminate as true , is due to ShuffleRdd not taking into account , as the ShuffleDependency is not taking into account inDeterminate nature of HashPartitioner. Moreover the attribute ref provided to HashPartitioner though is pointing to an inDeterminate alias, is having deterministic flag as true ( which in itself is logically correct). So I feel apart from ShuffleDependency code change, all expressions should have a lazy Boolean say containsIndeterministic component. Which will be true if the expression is indeterministic or contains any attribute ref which has is indeterministicComponent true.
And on personal note.. thanks for your interest.. this is very rare attitude. Regards Asif On Sun, Jan 26, 2025, 9:45 PM Ángel <angel.alvarez.pas...@gmail.com> wrote: > Hi Asif, > > Could you provide an example (code+dataset) to analize this? Looks > interesting ... > > > Regards, > Ángel > > El dom, 26 ene 2025 a las 20:58, Asif Shahid (<asif.sha...@gmail.com>) > escribió: > >> Hi, >> On further thoughts, I concur that leaf expressions like AttributeRefs >> can always be considered to be deterministic, as , as a java variable the >> value contained in it per iteration is invariant ( except when changed by >> some deterministic logic). So in that sense what I said in the above mail >> as that an issue is incorrect. >> But I think that AttributeRef should have a boolean method which tells, >> whether the value it represents is from an indeterminate source or not. >> Regards >> Asif >> >> >> >> On Fri, Jan 24, 2025 at 5:18 PM Asif Shahid <asif.sha...@gmail.com> >> wrote: >> >>> Hi, >>> While testing a use case where the query had an outer join such that >>> joining key of left outer table either had a valid value or a random value( >>> salting to avoid skew). >>> The case was reported to have incorrect results in case of node failure, >>> with retry. >>> On debugging the code, have found following, which has left me confused >>> as to what is spark's strategy for indeterministic fields. >>> Some serious issues are : >>> 1) All the leaf expressions, like AttributeReference are always >>> considered deterministic. Which means if an attribute is pointing to an >>> Alias which itself is indeterministic, the attribute will still be >>> considered deterministic >>> 2) In CheckAnalysis there is code which checks whether each Operator >>> either supports indeterministic value or not . Join is not included in the >>> list of supported, but it passes even if the joining key is pointing to an >>> indeterministic alias. ( When I tried fixing it, found a plethora of >>> operators failing Like DeserializedObject, LocalRelation etc which are not >>> supposed to contain indeterministic attributes ( because they are not in >>> the list of supporting operators). >>> 3) The ShuffleDependency does not check for indeterministic nature of >>> partitioner ( fixed it locally and then realized that there is the bug #1 >>> which needs to be fixed too). >>> >>> The code in DagScheduler / TaskSet, TaskScheduler etc, seems to have >>> been written , keeping in mind the indeterministic nature of the previous >>> and current stages , so as to rexecute previous stages as a whole, instead >>> of just missing tasks, but the above 3 points, do not seem to support the >>> code of DagScheduler / TaskScheduler. >>> >>> Regards >>> Asif >>> >>>
