Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19938
Sure, in #19865, I will generate new parameter name in `splitExpression`
instead of inserting assertion.
---
-
To unsubscribe,
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19938
I prefer to generate new parameter name in `splitExpression` over
localizing global variables. There is no contract that an `Expression` must
output java variables, we may inline some values,
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/19938
Is there only the place where we need this localization (I mean other
operators don't need this logic)? I'm also neutral about this pr though, I feel
better to make this more general to avoid the
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19938
This makes sense to me.
Currently, based on previous discussion, we are fixing code generation and
insert an assertion at #19865 to ensure no global variables are passed.
Of course, as
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19938
Good point.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19938
actually there is one real problem: after we fold many global variables
into an array, the variable name may become something like `arr[1]`, which
can't be used as the parameter name.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19938
I have the same feeling with @viirya , expressions/operators usually won't
mutate input from children. One concern is how to guarantee this
programmatically, but might be OK as I don't think this
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19938
Thank you for great thought. Let me think about it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19938
They are possibly passed in split functions. But it is hard to image a case
we will change their values in the functions. In SparkSQL, the output from a
child operator are just used as input to
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19938
Can we guarantee 100% that any operation in `consume()` at
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19938
IIUC, the problem is only happened when we wrongly pass global variables
into split functions and change the values. Will we change the result variables
from aggregation?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19938
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19938
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84694/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19938
**[Test build #84694 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84694/testReport)**
for PR 19938 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19938
**[Test build #84694 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84694/testReport)**
for PR 19938 at commit
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19938
Jenkins, retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19938
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19938
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84691/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19938
**[Test build #84691 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84691/testReport)**
for PR 19938 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19938
**[Test build #84691 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84691/testReport)**
for PR 19938 at commit
20 matches
Mail list logo
