[GitHub] spark issue #23252: [SPARK-26239] File-based secret key loading for SASL.

2018-12-07 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23252 How is this file protected in kubernetes? I actually don't like this idea at least for yarn and other deployments, I see people abusing it (accidentally) and using it in non-secure m

[GitHub] spark issue #23242: [SPARK-26285][CORE] accumulator metrics sources for Long...

2018-12-07 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23242 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 if you aren't seeing actual issues with this I guess it would be interesting to test it further to see if it does. I can see spark blacklisting when it shouldn't for exit codes like y

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-05 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 the approach here makes sense. Are you seeing actual issues with this blacklisting when it shouldn't? I could see that possible there and if so we should move this to defect and make su

[GitHub] spark pull request #23223: [SPARK-26269][YARN]Yarnallocator should have same...

2018-12-05 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23223#discussion_r239110361 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -612,11 +612,14 @@ private[yarn] class

[GitHub] spark issue #23223: [SPARK-26269][YARN]Yarnallocator should have same blackl...

2018-12-05 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23223 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 thanks @redsanket @squito, committed master, 2.4, and 2.3.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-30 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237875851 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala --- @@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var

[GitHub] spark issue #23160: [SPARK-26196][WebUI] Total tasks title in the stage page...

2018-11-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23160 reviewing, to clarify @pgandhi999 question, I assume the screenshots above are cut off and the task table is really showing 7 total failed tasks

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-28 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237248890 --- Diff: python/pyspark/broadcast.py --- @@ -134,7 +137,15 @@ def value(self): """ Return the br

[GitHub] spark pull request #23166: [SPARK-26201] Fix python broadcast with encryptio...

2018-11-28 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/23166#discussion_r237212353 --- Diff: python/pyspark/tests/test_broadcast.py --- @@ -67,6 +67,20 @@ def test_broadcast_with_encryption(self): def

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 Yeah certainly seems like a good idea. The only question I have is does this cause more memory usage on the driver because it has a reference to that broadcast value or is something else already

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 cc @squito --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23166: [SPARK-26201] Fix python broadcast with encryption

2018-11-28 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23166 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 +1 , going to merge to master There are a few followup jiras on this. 1) make the timeline visualization better: https://issues.apache.org/jira/browse/SPARK-26130 2) improve

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 Test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23103 Lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 Test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23103 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r235164507 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -92,6 +92,14 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-20 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r235157033 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -259,7 +278,8 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark issue #19840: [SPARK-22640][PYSPARK][YARN]switch python exec on execut...

2018-11-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/19840 I didn't read the entire thread here but what you want is this: --archives hdfs:///python36/python36.tgz#python36 --conf spark.pyspark.python=./python36/bin/python3.6 -

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r234765781 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -315,187 +241,22 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r234757635 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S

2018-11-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23013 sorry for my delay as I was on vacation. I would like to clarify if k8s support spark.authenticate and if so how does it do it?Without that, Spark is not secure with k8s unless you are

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232832492 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -162,26 +165,29 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232738700 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736598 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736621 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232736375 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232735802 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-11-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r232711679 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,120 @@ private[v1] class StagesResource extends

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-11-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 I think we should add the support back. It sounded like some people didn't like this PR for a fix so we would need to investigate something else, @cloud-fan did you have more specifics

[GitHub] spark issue #23013: [SPARK-25023] More detailed security guidance for K8S

2018-11-12 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/23013 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-11-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 merged to master and 2.4 branch, thanks @dhruve --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-11-06 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-11-02 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r230487066 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +507,181 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-11-02 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 merged to master and 2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

2018-11-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21688 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-10-31 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 @dhruve is this ready to review again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-31 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 If no other comments, I'll commit this? I'll leave it open for a bit longer --- - To unsubscribe, e-mail: review

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-30 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r229363078 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,959 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 added sections to the resource manager sections. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I can add a note here for deployments here and then we can do version specific ones after --- - To unsubscribe, e-mail

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-30 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 Updated to have a section on security in the quickstart and overview, let me know what you think and if wording needs updated. If this ok I can followup with something on the website

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r229002442 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r228994601 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-29 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r228982925 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,944 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would be fine with adding it more places, including perhaps the overview page: http://spark.apache.org/docs/latest/ and quick start pages. Perhaps we should agree upon the wording here

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-29 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21588 Can you clarify what you mean by drop builtin metastore support? Are you just saying users must always provide jars to use it or something more

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r228678209 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 I would rather see someone more familiar with K8s that uses it document it. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 so I had filed a jira to update mesos docs more detail about security things (https://issues.apache.org/jira/browse/SPARK-25024) which I need to follow up on, but I didn't file one for k8s

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r228668756 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 we can add stronger wording for standalone if you want, I know the text was recently updated (I believe by you) to have the below: > For other resource manag

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 the intention is not a we told you so, its meant to grab their attention and to get people to think about it because in the end it is their responsibility in my opinion. I'm fine i

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22852#discussion_r228544730 --- Diff: docs/security.md --- @@ -6,7 +6,20 @@ title: Security * This will become a table of contents (this text will be scraped). {:toc

[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation

2018-10-26 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22852 @vanzin @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22852: [SPARK-25023] Clarify Spark security documentatio...

2018-10-26 Thread tgravescs
GitHub user tgravescs opened a pull request: https://github.com/apache/spark/pull/22852 [SPARK-25023] Clarify Spark security documentation ## What changes were proposed in this pull request? Clarify documentation about security. ## How was this patch tested

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-25 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 no other comments, looks good. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-24 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 @cloud-fan I agree with you that most issues do not have to block a release but I don't agree with most of your criteria for making that decision. I've already talked about

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-24 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 while I'm ok with not blocking 2.4 for this as well, not for many of the reasons stated though. Note the jira was filed a Major not a blocker. Based on the information we have, the impact o

[GitHub] spark pull request #22771: [SPARK-25773][Core]Cancel zombie tasks in a resul...

2018-10-23 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22771#discussion_r227503789 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1364,6 +1385,19 @@ private[spark] class DAGScheduler

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 The fact that it isn't a new regression shouldn't determine if its a blocker, there are lots of things you don't hit right away. The impact does affect if we decide if th

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 @markhamstra thanks for the reference, I was looking for some background on this. I agree those are still issues like mentioned in SPARK-17064 but I don't think that directly impacts this

[GitHub] spark issue #22144: [SPARK-24935][SQL] : Problem with Executing Hive UDF's f...

2018-10-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22144 @cloud-fan I disagree, it seems like you broke functionality that was there and you can't do that in a minor release. 3.0 would be fine to drop I think but we should fix it for 2.4, th

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227117575 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -547,6 +519,24 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227116341 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r227116251 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -52,22 +52,20 @@ $.extend( $.fn.dataTable.ext.type.order

[GitHub] spark pull request #22504: [SPARK-25118][Submit] Persist Driver Logs in Clie...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22504#discussion_r227036915 --- Diff: docs/configuration.md --- @@ -266,6 +266,41 @@ of the most common options to set are: Only has effect in Spark standalone mode or Mesos

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r226999715 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r227005479 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +505,145 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r226999389 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -453,6 +503,22 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-22 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r227000658 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,54 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new execut...

2018-10-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22288 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

2018-10-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 Actually as part of the jiras I mentioned above we were looking at killing other task attempts as soon as one task attempt succeeds rather then waiting for the entire job to finish, thoughts on

[GitHub] spark pull request #22771: [SPARK-25773][Core]Cancel zombie tasks in a resul...

2018-10-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22771#discussion_r226651308 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1364,6 +1385,16 @@ private[spark] class DAGScheduler

[GitHub] spark issue #22771: [SPARK-25773][Core]Cancel zombie tasks in a result stage...

2018-10-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22771 the change itself seems reasonable we were actually looking at this recently as there are a couple of others bugs around this: https://issues.apache.org/jira/browse/SPARK-25250 and https

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-17 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225998426 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +420,65 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #22725: [SPARK-25753][[CORE][FOLLOW-UP]fix reading small files v...

2018-10-17 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 +1 Looks good, thanks @10110346 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

2018-10-16 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/22725 SPARK-24610 is the original issue, please file a new jira for StreamFileInputFormat --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225198423 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -503,6 +505,89 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225193634 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -415,9 +419,61 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

2018-10-15 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/22288#discussion_r225185389 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -597,6 +597,16 @@ package object config { .checkValue(v

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224864545 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-12 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224850397 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224251226 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -278,7 +198,7 @@ private[ui] class StagePage(parent: StagesTab, store

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224239984 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala --- @@ -102,4 +103,124 @@ private[v1] class StagesResource extends

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224238974 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -341,7 +341,9 @@ private class LiveExecutorStageSummary

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224221297 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -349,7 +349,23 @@ private[spark] class AppStatusStore( def

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224217796 --- Diff: core/src/main/resources/org/apache/spark/ui/static/utils.js --- @@ -74,3 +74,99 @@ function getTimeZone() { return new Date().toString

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224178567 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224176616 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html --- @@ -0,0 +1,124 @@ + + +

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224166110 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224164134 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224163278 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224162695 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

2018-10-10 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21688#discussion_r224162463 --- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js --- @@ -0,0 +1,872 @@ +/* + * Licensed to the Apache Software

  1   2   3   4   5   6   7   8   9   10   >