Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
Here is a PR introduces `maxFields` parameter to all function involved in
creation of truncated strings of spark plans:
https://github.com/apache/spark/pull/23159
---
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
I am closing the PR since a part of it has been merged in #23018 already,
and the rest is coming soon.
---
-
To unsubscribe,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
Ooops i rished to read. Yea but still sounds related but orthogonal. Let's
move it to mailing list. That should be the best place to discuss further.
---
Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
> @boy-uber, for structured streaming, let's do it out of this PR. I think
the actual change of this PR can be small (1.). We can change this API for
structured streaming later if needed since
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
@boy-uber, for structured streaming, let's do it out of this PR. I think
the actual change of this PR can be small (1.). We can change this API for
structured streaming later if needed since
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@HyukjinKwon @cloud-fan Thank you for looking at the PR.
So, if I split the PR to 2 PRs:
1. Writing truncated plans to a file
2. Control number of fields in truncated strings.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
I took a super quick pass - the change actually quite looks okay in general
to me.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
@MaxGekk, couple of questions for its implementation from a cursory look.
It's the implementation is complicated here:
1. it tries to use writer and avoid to construct the
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@cloud-fan
> This is hard to review, do you mean we should add maxFields: Option[Int]
to all the string related methods?
Not to all but only to methods involved to producing
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22429
This is hard to review, do you mean we should add `maxFields: Option[Int]`
to all the string related methods?
---
-
To
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@cloud-fan @gatorsmile May I ask you to look at the PR. It stuck for a
while by unclear reasons but I believe the proposed method `toFile` could be
pretty useful in troubleshooting different
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@hvanhovell Could you look at the PR, please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98475/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98475 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98475/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98475 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98475/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98465/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98465 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98465/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98465 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98465/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98461/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98461 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98461/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98461 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98461/testReport)**
for PR 22429 at commit
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
jenkins, retest this, please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
> > @MaxGekk I sent email to spark dev list about structured plan logging,
but did not get any response.
>
> @boy-uber I guess It is better to speak about the feature to @bogdanrdc
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@gatorsmile @HyukjinKwon @viirya @rednaxelafx Are you ok with the proposed
changes or there is something which blocks the PR for now?
---
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
May I ask you @hvanhovell @zsxwing to review the PR one more time.
---
-
To unsubscribe, e-mail:
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
> @MaxGekk I sent email to spark dev list about structured plan logging,
but did not get any response.
@boy-uber I guess It is better to speak about the feature to @bogdanrdc
@hvanhovell
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98324/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98324 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98324/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #98324 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98324/testReport)**
for PR 22429 at commit
Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
> > @boy-uber the thing you are suggesting is a pretty big undertaking and
beyond the scope of this PR.
> > If you are going to add structured plans to the explain output, you
probably also
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97600/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97600 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97600/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97600 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97600/testReport)**
for PR 22429 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97580/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97580 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97580/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97580 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97580/testReport)**
for PR 22429 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
I am able to address his comments for his vacation. Please keep reviewing
this.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97302/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97302 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97302/testReport)**
for PR 22429 at commit
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@hvanhovell @zsxwing Could you look at this during the next a few days
otherwise I will be able to come back to the PR in 3 weeks, please.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97302 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97302/testReport)**
for PR 22429 at commit
Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
> @boy-uber the thing you are suggesting is a pretty big undertaking and
beyond the scope of this PR.
>
> If you are going to add structured plans to the explain output, you
probably also
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97253/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97253 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97253/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97256/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97256 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97256/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97256 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97256/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #97253 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97253/testReport)**
for PR 22429 at commit
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/22429
@boy-uber the thing you are suggesting is a pretty big undertaking and
beyond the scope of this PR.
If you are going to add structured plans to the explain output, you
probably also
Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
Great to see the code here to capture more information from the plan and
save to file. We have similar needs with our Spark applications as well.
Adding my two cents:
1. Anyone
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@zsxwing Please, have a look at the PR one more time.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
Is there anything which blocks the PR for now?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@gatorsmile @zsxwing @hvanhovell @viirya @rednaxelafx @HyukjinKwon Please,
take a look at the PR.
---
-
To unsubscribe,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96718/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96718 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96718/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96718 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96718/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96713/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96713 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96713/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96713 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96713/testReport)**
for PR 22429 at commit
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
As far as I know, @hvanhovell, you would prefer replacing `Option[Int]` by
just `Int` in `simpleString` and other similar places. I can do that but I have
to read the config parameter in all those
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96551/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96551 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96551/testReport)**
for PR 22429 at commit
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
> Could you also try to remove the default value?
I have removed the default value `None` but still keep type as
`Option[Int]`. @zsxwing Is it ok?
>
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96551 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96551/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96527/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96527 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96527/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96527 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96527/testReport)**
for PR 22429 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/22429
@MaxGekk Make sense. Could you also try to remove the default value?
---
-
To unsubscribe, e-mail:
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
> Can we change `maxFields: Option[Int]` to `maxFields: Int` so that the
caller will never forget to pass it in and it should either get it from some
place or use the default value.
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@gatorsmile @rednaxelafx @HyukjinKwon @viirya @hvanhovell Could you review
the PR, please.
---
-
To unsubscribe, e-mail:
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
@hvanhovell May I ask you to look at this one more time.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96214/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96214 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96214/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96214 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96214/testReport)**
for PR 22429 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22429
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96192/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96192 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96192/testReport)**
for PR 22429 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96192 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96192/testReport)**
for PR 22429 at commit
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/22429
Tests failed on `PythonForeachWriterSuite`. Waiting for
https://github.com/apache/spark/pull/22452
---
-
To unsubscribe,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96178/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22429
**[Test build #96178 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96178/testReport)**
for PR 22429 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22429
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96175/
Test FAILed.
---
1 - 100 of 118 matches
Mail list logo
