[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 Thank you everybody for their valuable comments. I do agree with Tom that optimizations need to be done on this issue. We need to move sorting to server side and try reducing the number of loops used to parse the json files and a couple of other things. So I shall close this PR for the time being. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 After reading through Tom and Marcello's comments I'll defer to them on whether this should be merged in or not. I will note that moving from a hard-coded, server-side, scala generated web ui to a dynamic, client-side, js generated web ui utilizing the metrics api is the direction we've been moving the ui for some time, first with the SHS summery page then with the executors page. I still think this is the right direction, but of all the remaining pages to convert this page at the same time has both the most to gain and the most potential to drag down the UI. If you guys think holding off on this until Marcello's work is done and re-optimizing then is better I'll stand by that, and even though I never did heavy testing for speed I'll stand by Tom that this still needs optimization work before merging. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 ok trying this out it is taking a lot longer then I expected. retrieving the data is relatively quick (2 seconds for 4 tasks) but then is taking a lot longer to parse and then display. Total time for 4 was 17 seconds. Looking at the code itself we seem to be doing a lot of loops over the data. If we can fix that to only make one pass over it, it would be a lot better. Or we change to do server side. Either way it sounds like this needs more work and if we have other jira with large changes perhaps we should hold off on this until some of that gets in. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 seems to be issues with the pr still. Getting errors trying to run on larger jobs: project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:347) at scala.None$.get(Option.scala:345) at org.apache.spark.status.api.v1.AllStagesResource$.convertTaskData(AllStagesResource.scala:197) at org.apache.spark.status.api.v1.AllStagesResource$$anonfun$3.apply(AllStagesResource.scala:76) at org.apache.spark.status.api.v1.AllStagesResource$$anonfun$3.apply(AllStagesResource.scala:76) Also I see the accumulators table on jobs that shouldn't have it, this was on a running application. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 disclaimer is I personally haven't tried this out on larger stages so I'll try to do that tomorrow to really see how the user experience is with larger # of tasks. If its really that bad then definitely agree doing the server side stuff makes sense. I wouldn't expect it to take 55 seconds. if its really taking that long seems like something would be wrong. especially on a running application or after initial history load. I was expecting the change even for large applications to better then the current if you are actually doing something more then the very basic. Sorry I've unfortunately been busy doing internal projects so I haven't had time to help with many reviews. It should get better in a week or two and I'll try to look at yours prs to help. I also didn't think the initial part of that jira was doing anything with the UI so didn't expect these things to conflict and of course this has taken a lot longer then expected to get up too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 > from my understanding SPARK-20657 is proposing to not using the rest api Not really; I just don't know enough JS to actually make the change to use the REST api, but I do believe using it is a better approach. The main thing about that change is to generate the view the client wants without having the whole data set in memory (either in the client or server side). > the question is do we want to do it now with this pr, or... I'm not really against the idea of the changes in this PR, but if we're changing things, I'd like at least a little more thought put into how to do things right. I don't think the "transfer everything from the server to the client" approach is the right thing to do in the long term; it will make looking at small stages better, yes, but it will make looking at larger stages worse. Even if afterwards it's quicker to resort columns, I have little patience to wait 1min for a page to load. And so do not others who have filed bugs about that kind of thing. Another point I'd like to make, and this is selfish me talking, is that large changes in these areas really slow down progress in SPARK-18085. Everybody tries to make local fixes to the problems I've tried to holistically fix in that project, and very few people seem interested in reviewing the code I've been putting out. I believe the changes in that project will make changes like this a lot easier, so I'd really appreciate getting those changes reviewed and committed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 Right so from my understanding https://issues.apache.org/jira/browse/SPARK-20657 is proposing to not using the rest api and force more on the backing store which exists only for history server but not a running application. Running application already have this data in memory anyway. Or are you proposing changing the rest api to use the new backing store and the stage page to the rest api? If not then you are splitting the ui pages to be more specific to running vs history or at least having a different backend store to fetch them from. If its specific to the ui pages then it doesn't help the rest api. If it helps the rest api then why not use that for the web pages too. Either way the data has to get to the web browser, I would think not using the rest api would be memory efficient if it can read directly from existing objects (or the backing store) without having to create a new objects in the rest api to send out. Maybe that doesn't matter so much if we make it more server side to send out small bits at a time. The rest api would be a nice way to abstract the backend out from the UI pages but if it doesn't perform well enough we shouldn't do it. Either one is doable I think we should just choose a direction. I think the server side stuff makes a lot of sense the question is do we want to do it now with this pr, or perhaps a pre-jira to this one, or do we do it later. To me the current pages are frustrating enough I don't mind doing it later but it probably depends on how often you load one of the pages with 100,000+ tasks. I don't think its any slower then loading the history server pages now (which I'm glad you are working on) Let me know your thoughts. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82693/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82693/testReport)** for PR 19270 at commit [`ef90c2a`](https://github.com/apache/spark/commit/ef90c2a2c5c95daef7ce04e0c348181b2e18f677). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 The main change I'm talking about is https://issues.apache.org/jira/browse/SPARK-20657 (code at https://github.com/vanzin/spark/pull/41). I did not change the format of the tables, but how data is loaded into them. The new backing store is much faster at sorting than the current code; and it tries to only load the data that will be shown, so it's also light on memory. The only think it doesn't support, as I mentioned, is searching, but that could be added at a cost. (And it currently doesn't cache metrics, so each page load scans all metrics which is a little slow, but an order of magnitude faster than the numbers shown above.) I'm not a fan of the tables currently, nor an I saying they should stay as is. But my main concern here is really SHS memory usage. Your point of hitting the rest api is valid, and I think it should be considered a bug that no default limit is imposed for large lists. The way we've generally solved the "arbitrarily large" lists in other apps here is to use infinite scrolling + server side search. I'm not really a front end dev so I don't know what the current opinions on that approach is, but it's definitely lighter as far as memory usage on the server side goes, and load times on the client side are pretty fast too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 @vanzin I haven't followed the prs from SPARK-18085 closely, you keep mentioning it here but don't give anymore context as to what that has to do with this change. Did you change the format of tables in there? Please point to specifics. Yes there is obviously a trade off here since its loading all the data at first. This is useful in many situations but not in others. the memory usage can be an issue but isn't worse then anyone hitting the rest api directly (other then your browser side).Personally I hate the tables as is now, they are not friendly at all when looking for things in the task tables so we need to improve them somehow. If you have other design ideas for this perhaps you can talk about those. There are definitely other approaches/frameworks, we can definitely improve upon this to make things happen more on the server side if its needed and we support in the rest api. We could also change back away from the rest api, but I thought that is the way we were wanting to go but maybe that has changed again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 > sort over all the entries rather than the 20 entries in the current page Sort is over all entries, isn't it? If it isn't, the work I have on SPARK-18085 sorts over all entries and is pretty fast at that. > the loading time is approximately 50-55 That's pretty slow. My code in SPARK-18085 takes a few seconds to load each page, and can be sped up a lot by caching some data that has to be computed on every load. It doesn't support search, but that can be added. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 > Does datatables support requesting pages dynamically from the server side instead of having to load the full data set from the get go? If that is the case, then my change serves no purpose. As I have mentioned in the PR, we would like to support sort over all applications rather than the 20 entries in the current page and this is not possible if the data is loaded dynamically. > But how high is high? For around 10 tasks as you suggested in the above comment, the loading time is approximately 50-55 seconds which is compensated by the extremely short times for sorting, searching and pagination for a hundred thousand tasks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 Also, there's a second question of how much memory does it take to generate the JSON for such a large stage. While working on SPARK-18085 I remember measuring this and one time requesting the stage page caused more than 600MB of memory to be used (which caused my small driver to OOM). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 > the initial page loading time may be slightly high But how high is high? Does datatables support requesting pages dynamically from the server side instead of having to load the full data set from the get go? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82693/testReport)** for PR 19270 at commit [`ef90c2a`](https://github.com/apache/spark/commit/ef90c2a2c5c95daef7ce04e0c348181b2e18f677). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @vanzin Actually the JIRA was reported by somebody else and I copied the description in the first para. so I just knew what needed to be done but missed out on the specifics of the description. I have updated the PR description. Thank you for pointing that out. Regarding your comment on the speed of JS execution for really large stages, the initial page loading time may be slightly high, but it is greatly compensated by the time spent in retrieving subsequent task info in the data table as opposed to the traditional HTML table, where time spent retrieving task information for every 10 tasks or so is higher. And also the initial page loading time overhead is compensated for the convenience that the Search functionality provides across all columns and the speed with which it supports sorting of the data. So, in the end, it boils down to how useful this change is as opposed to the cost involved in implementing the change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19270 I'm confused about the PR description. It mentions "sorting was disabled if there is any application that has more than one attempt" which as far as I know has nothing to do with the stage page. Also what's a "Stage Server" (from "sort and search for Stage Server")? As far as the change itself, I assume it's only related to the stage page, and from the little javascript I know, you seem to be downloading the whole stage with all its tasks from the REST API. How fast is that with really large stages (e.g. 100k tasks which I think is the default retention)? You can try `sc.parallelize(1 to 10, 10).count()` to create a large stage. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82646/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82646/testReport)** for PR 19270 at commit [`f649e42`](https://github.com/apache/spark/commit/f649e42aa01de9da4dd7747a069a34ae99e49e30). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82646/testReport)** for PR 19270 at commit [`f649e42`](https://github.com/apache/spark/commit/f649e42aa01de9da4dd7747a069a34ae99e49e30). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth I do not quite understand what you are saying. Everything seems to be working fine on my test setup. Can you please let me know how do I replicate the issue? Thank you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 So I think I know why the appId was handled the way it was, the live app ui no longer works because the appId var is "undefined" in all the api calls --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82563/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82563/testReport)** for PR 19270 at commit [`0b2a8cf`](https://github.com/apache/spark/commit/0b2a8cfaab8fa6bcb92176f74dce2f47ba65454d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 Yep I did that! Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Also could you update the description with new (and more) screen shots? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Thanks, I'll take a look later today --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth Fixed the issue with the accumulators table. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82563/testReport)** for PR 19270 at commit [`0b2a8cf`](https://github.com/apache/spark/commit/0b2a8cfaab8fa6bcb92176f74dce2f47ba65454d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82522/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82522/testReport)** for PR 19270 at commit [`25b5215`](https://github.com/apache/spark/commit/25b5215cd6dbbc11b5e5d8d56bff6743578de2b5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 Ok Yes, I just noticed that comment. Will get back to you on this one ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Thanks, I must've missed that in the description. You've take care of all but my last comment, I'm not seeing the accumulators table, have you checked it shows up when theres accumulators? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth Regarding your earlier comment on the missing functionality of show additional metrics, I have mentioned that in the description of the PR. I restate it below: Because of the above change, certain functionalities in the page had to be modified to support the addition of datatables. For example, the toggle checkbox 'Select All' previously would add the checked fields as columns in the Task table and as rows in the Summary Metrics table, but after the change, only columns are added in the Task Table as it got tricky to add rows dynamically in the datatables. I have fixed the second bit of missing functionality about creating link on completed tasks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82522/testReport)** for PR 19270 at commit [`25b5215`](https://github.com/apache/spark/commit/25b5215cd6dbbc11b5e5d8d56bff6743578de2b5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 No problem, @ajbozarth , and thank you for your valuable feedback. I really appreciate it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Thanks, I'll pull the latest changes and keep testing. And thanks for your quick responses, I understand large changes like this take forever to review and can get frustrating for the submitter. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth I have fixed the issue of my changes not working in the web ui for local, standalone and yarn. Let me know if you are still facing issues with the testing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82321/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82321/testReport)** for PR 19270 at commit [`c1f85ae`](https://github.com/apache/spark/commit/c1f85aeab03e2d172261431d6e5e46cbd303e069). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82321/testReport)** for PR 19270 at commit [`c1f85ae`](https://github.com/apache/spark/commit/c1f85aeab03e2d172261431d6e5e46cbd303e069). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth There were two unit tests in StagePageSuite.scala that were failing as they are no longer valid for the modified ui that generate datatables dynamically from Javascript. I have removed them for the time being, but if you have any other suggestions, let me know. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82252/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82252/testReport)** for PR 19270 at commit [`098a93d`](https://github.com/apache/spark/commit/098a93d9c8e2cc31ef3b43c11d653c2e58da298a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82252/testReport)** for PR 19270 at commit [`098a93d`](https://github.com/apache/spark/commit/098a93d9c8e2cc31ef3b43c11d653c2e58da298a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 Ok, I will look into it. I am currently fixing ui bugs and unit tests, so will commit those changes first, then will look into the above issue. Thank you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Ok so I'm still doing more testing but I've narrowed the above problem. The above error is occurring when using either local or standalone, the error doesn't appear when using yarn. I'll continue my testing and review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 I just tried this out and it appears to be working for me for a running application, haven't tried the history UI yet.@ajbozarth What browser are you using and what are you running (a wordcount, or similar)? I tried both in chrome and firefox. I just checked out his pull request and built that. one thing I noticed was if you select everything under the "Show additional metrics", then refresh the page, is saves the settings but the check boxes aren't checked anymore. "Select All" should be called what it used to be "(De)select All" It also seems to be missing some of the options there when you do a shuffle there should be Shuffle read/write metrics: "Shuffle Read Blocked Time" and "Shuffle Remote Reads" --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270  --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 For me it's definitely the UI that doesn't work and the SHS that does, I'' see if I can recreate and screenshot the js error I'm getting for you --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 I believe you mean the opposite of what you wrote. My changes are visible in the web ui(while the app is running) and not in the SHS(once the job is done). Yep I see that and am working on the fix. Thank you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 I'm still going through the code but I also checked out, built and ran you changes and found that the page doesn't work in the web UI only in the SHS. Did you test this on both the Web UI and SHS? I'll continue my read through and testing of your code while you fix this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82003/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82003/testReport)** for PR 19270 at commit [`dd12be7`](https://github.com/apache/spark/commit/dd12be7db42a4de134d12dfd98dacddb074af4b8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #82003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82003/testReport)** for PR 19270 at commit [`dd12be7`](https://github.com/apache/spark/commit/dd12be7db42a4de134d12dfd98dacddb074af4b8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 No problem. Thank you for your valuable comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 On a second look I think I figured out my misunderstanding, and I've realized a through review will take quite a bit of time, I'll do my best to finish by the end of the week but no promises. As for the MiMa failure, any change to a public api (even additions) must be added to the MiMa excludes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 The error logs for test build #81683 state that method this(Long,Int,Int,Long,Long,Long,Long,Long,Long)Unit in class org.apache.spark.status.api.v1.ExecutorStageSummary does not have a correspondent in current version. All I have done is add new fields in the api ExecutorStageSummary and have not modified any existing ones. It should be fine but please let me know if it is not. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81936/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #81936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81936/testReport)** for PR 19270 at commit [`c588953`](https://github.com/apache/spark/commit/c58895357d568b5980a8cda7420536c3d87af4c5). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19270 **[Test build #81936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81936/testReport)** for PR 19270 at commit [`c588953`](https://github.com/apache/spark/commit/c58895357d568b5980a8cda7420536c3d87af4c5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19270 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 All of the code is part of the same task. Can you please be more specific about the code that you have doubts about, and I can elaborate further on it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 I'll look at the html/js code tomorrow, but it looks like there still unrelated code that adds new fields, is that code supposed to be there or is it for another task? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19270 Thanks, I'll try to review this by EOD tomorrow --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user pgandhi999 commented on the issue: https://github.com/apache/spark/pull/19270 @ajbozarth Thank you for your comment on the previous PR. I have closed that one. Apologies for the confusion caused in the previous PR! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19270: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19270 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
