Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
I am closing this as there is still some discussion to go on in order to
choose the right approach. We can reopen it later/open a new one once we choose
the approach. Thanks.
---
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
@maropu this is a bit stuck because we first need to decide how to go on.
Above you can see my analysis and comparison with other tool and now we have to
choose what to do before going on. If you
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/20692
@mgaido91 Can you update this? Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
sure, @gatorsmile, no problem. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
@mgaido91 Thanks for you investigation! These two weeks I am swamped. Will
get back to you next week. Sorry for the delay.
---
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
any thoughts @cloud-fan @dongjoon-hyun @gatorsmile @rdblue about the above
analysis? Thanks.
---
-
To unsubscribe, e-mail:
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
thanks @gatorsmile and @rdblue for your discussion and thanks @gatorsmile
for trusting me. I hope I will be worthy of your trust.
I agree that since we are not close to a new release,
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
Good point on nested types. I don't think heavy nesting is the usual case,
but we can definitely improve the explain result in the long term by separating
it out. Maybe just using a high-level type
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20692
BTW I think we do need a story to improve the explain result. The current
explain works pretty well with simple queries. But for complex queries, that a
single plan node across many lines, it's
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20692
My 2 cents: a data source may have a lot of columns, and a column may be a
complex struct type. So embedding the type in the plan may make the plan
unreadable.
If I'm debugging and see a
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
My intent is just to advocate for clear feedback on the content of PRs.
Good to hear your confidence in @mgaido91, and if he wants to work on a better
explain, that's great too.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
Thanks for your suggestion. I think different communities have different
cultures. For new contributors, we always try to encourage them and assign them
some trivial JIRAs and fixes.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
> I think you can add the patch to your fork.
My primary concern isn't the feature, although I do think it is useful. My
concern is how we work with contributors. My worry here is that the
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
Normally, in our Spark community, if the contributor is not available when
we decide to merge, the others (including the committers) will take it over and
give the credit back the contributor.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
> this is not urgent based on our release schedule.
Marco is contributing this right now. It is a bad idea to ask contributors
to show up in 4 months, if we don't have a better option by
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
First, this is not urgent based on our release schedule. If no better
solution is proposed, we can still revisit this PR before the release.
Second, adding more extra string functions
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87847/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87847 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87847/testReport)**
for PR 20692 at commit
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
I don't think it is a good idea to block a fix like this on a redesign of
explain plans. If that redesign was currently being worked on and this made it
more difficult, it might be reasonable. But
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
The current solution looks not clean to me. I would like to hold it first
since the next release is after another 6 months.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
If possible, we need the investigation first and then need more discussions
before finalizing the design.
---
-
To
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
I don't see a compelling reason to block this work, which is well-defined
and a reasonably small patch. What is here would help users with debugging.
@gatorsmile, If you have an alternative
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
@gatorsmile, yes, sure. Do you want me to investigate their behavior only
related to how they show data types or more in general?
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
@rdblue The types are important, but we do not need to embed it in the
plan. Each attribute has the unique identifier. How about outputting the
operator details after the plan? We need a clean
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20692
@mgaido91 It sounds like you are interested in the work on improvement of
EXPLAIN. Maybe we can first hold this PR and do the design first? Could you
investigate how the other DBs present their
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
@mgaido91, I'm happy without another option. Let's add it in the future if
we find that we need it, but not assume that we will. This change should give
most of the information needed to debug
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
thanks @rdblue, I updated the PR and now only the leaf nodes contain the
data type information.
Honestly I would avoid to have another option, since we already have the
extended one. I
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20692
I agree with @cloud-fan that the type information doesn't need to be in
every node of the plan, just in the scans and generators. We want enough
information that we can tell what types are there,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1198/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87847 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87847/testReport)**
for PR 20692 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
@cloud-fan I think it is not needed a new debug mode for it. I think it is
good to have the attribute type always, but in case I think we can re-use the
extended falg, without introducing a new
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20692
We should clearly define when and where we need to display attribute data
type. I think leaf nodes and some nodes that produce new data like `Generate`
are good places. And we may also need to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87801/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87801 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87801/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87800/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87800 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87800/testReport)**
for PR 20692 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87801 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87801/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1165/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20692
The failures of `SQLQueryTestSuite` are legitimate.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87791/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87791 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87791/testReport)**
for PR 20692 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87800 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87800/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1163/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1157/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87791 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87791/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20692
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87781/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87781 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87781/testReport)**
for PR 20692 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87781 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87781/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1150/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87773/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87773 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87773/testReport)**
for PR 20692 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20692
**[Test build #87773 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87773/testReport)**
for PR 20692 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20692
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1145/
66 matches
Mail list logo
