[jira] [Commented] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-09-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623660#comment-16623660 ] Thomas Graves commented on SPARK-24355: --- pr that got merged didn't get linked prop

[jira] [Assigned] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-09-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24355: - Assignee: Sanket Chintapalli > Improve Spark shuffle server responsiveness to non-Chunk

[jira] [Updated] (SPARK-25538) incorrect row counts after distinct()

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25538: -- Priority: Blocker (was: Major) > incorrect row counts after distinct() >

[jira] [Resolved] (SPARK-18364) Expose metrics for YarnShuffleService

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18364. --- Resolution: Fixed Fix Version/s: 2.5.0 > Expose metrics for YarnShuffleService >

[jira] [Assigned] (SPARK-18364) Expose metrics for YarnShuffleService

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-18364: - Assignee: Marek Simunek > Expose metrics for YarnShuffleService > -

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-10-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637240#comment-16637240 ] Thomas Graves commented on SPARK-25501: --- did you post SPIP to the dev list, I didn

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-10-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637248#comment-16637248 ] Thomas Graves commented on SPARK-25501: --- the spip title has "Structured Streaming"

[jira] [Assigned] (SPARK-25641) Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

2018-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25641: - Assignee: Sanket Reddy > Change the spark.shuffle.server.chunkFetchHandlerThreadsPercen

[jira] [Resolved] (SPARK-25641) Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

2018-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25641. --- Resolution: Fixed Fix Version/s: 3.0.0 > Change the spark.shuffle.server.chunkFetchHa

[jira] [Assigned] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24851: - Assignee: Parth Gandhi > Map a Stage ID to it's Associated Job ID in UI > -

[jira] [Resolved] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24851. --- Resolution: Fixed Fix Version/s: 3.0.0 > Map a Stage ID to it's Associated Job ID in

[jira] [Reopened] (SPARK-21809) Change Stage Page to use datatables to support sorting columns and searching

2018-10-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-21809: --- > Change Stage Page to use datatables to support sorting columns and searching > ---

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650560#comment-16650560 ] Thomas Graves commented on SPARK-25732: --- I would much rather see Spark start to pu

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651657#comment-16651657 ] Thomas Graves commented on SPARK-25732: --- So like Marcelo mentioned can't you re-us

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651827#comment-16651827 ] Thomas Graves commented on SPARK-25732: --- yeah I understand the concern, we don't w

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves commented on SPARK-25732: --- sorry just realized I misread the second

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:49 PM: -

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:53 PM: -

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657318#comment-16657318 ] Thomas Graves commented on SPARK-25692: --- [~redsanket] can you please take a look a

[jira] [Assigned] (SPARK-25753) binaryFiles broken for small files

2018-10-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25753: - Assignee: liuxian > binaryFiles broken for small files > --

[jira] [Resolved] (SPARK-25753) binaryFiles broken for small files

2018-10-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25753. --- Resolution: Fixed Fix Version/s: 3.0.0 > binaryFiles broken for small files > ---

[jira] [Commented] (SPARK-25855) Don't use Erasure Coding for event log files

2018-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665690#comment-16665690 ] Thomas Graves commented on SPARK-25855: --- it seems like it depends on whether you c

[jira] [Resolved] (SPARK-25023) Clarify Spark security documentation

2018-11-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25023. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > Clarify Spark secur

[jira] [Assigned] (SPARK-25023) Clarify Spark security documentation

2018-11-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25023: - Assignee: Thomas Graves > Clarify Spark security documentation > --

[jira] [Resolved] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15815. --- Resolution: Duplicate > Hang while enable blacklistExecutor and DynamicExecutorAllocator >

[jira] [Assigned] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22148: - Assignee: Dhruve Ashar > TaskSetManager.abortIfCompletelyBlacklisted should not abort w

[jira] [Resolved] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22148. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > TaskSetManager.abor

[jira] [Resolved] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17433. --- Resolution: Fixed Assignee: Thomas Graves Fix Version/s: 2.1.0 > YarnShuffleS

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484748#comment-15484748 ] Thomas Graves commented on SPARK-17321: --- Not sure I follow this comment. So you ar

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487203#comment-15487203 ] Thomas Graves commented on SPARK-17321: --- yes that makes sense and as I stated I thi

[jira] [Resolved] (SPARK-17511) Dynamic allocation race condition: Containers getting marked failed while releasing

2016-09-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17511. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-17582) Dead executors shouldn't show in the SparkUI

2016-09-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503461#comment-15503461 ] Thomas Graves commented on SPARK-17582: --- Yes they are meant to be there, status is

[jira] [Resolved] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16757. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > Set up caller

[jira] [Commented] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531056#comment-15531056 ] Thomas Graves commented on SPARK-16757: --- we are seeing issues with ReplSuite after

[jira] [Resolved] (SPARK-17710) ReplSuite fails with ClassCircularityError in master Maven builds

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17710. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > ReplSuite fai

[jira] [Updated] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17417: -- Affects Version/s: 1.6.0 2.0.0 > Fix sorting of part files while reconst

[jira] [Resolved] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17417. --- Resolution: Fixed Assignee: Dhruve Ashar Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585451#comment-15585451 ] Thomas Graves commented on SPARK-15708: --- [~ajbozarth] It was closed because we wer

[jira] [Commented] (SPARK-17979) Remove deprecated support for config SPARK_YARN_USER_ENV

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585664#comment-15585664 ] Thomas Graves commented on SPARK-17979: --- We already had a jira to remove all the de

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354336#comment-16354336 ] Thomas Graves commented on SPARK-23309: --- I pulled in that patch ([https://github.co

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-02-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1637#comment-1637 ] Thomas Graves commented on SPARK-22683: --- ok thanks,  I would like to try this out m

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-02-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355920#comment-16355920 ] Thomas Graves commented on SPARK-22683: --- If the config is set to 1 which keeps the

[jira] [Comment Edited] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-02-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356168#comment-16356168 ] Thomas Graves edited comment on SPARK-22683 at 2/7/18 10:24 PM: ---

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-02-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356168#comment-16356168 ] Thomas Graves commented on SPARK-22683: --- I agree, I think default behavior stays 1.

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-02-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357346#comment-16357346 ] Thomas Graves commented on SPARK-23309: --- sorry I haven't had time to make a query/d

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

2018-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383709#comment-16383709 ] Thomas Graves commented on SPARK-22479: --- [~aash] [~onursatici] this seems to have r

[jira] [Created] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

2018-03-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-23567: - Summary: spark.redaction.regex should not include user by default, docs not updated Key: SPARK-23567 URL: https://issues.apache.org/jira/browse/SPARK-23567 Project:

[jira] [Commented] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

2018-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383723#comment-16383723 ] Thomas Graves commented on SPARK-23567: --- I also question whether the url should be

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

2018-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383733#comment-16383733 ] Thomas Graves commented on SPARK-22479: --- Also the example above shows the password,

[jira] [Comment Edited] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

2018-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383723#comment-16383723 ] Thomas Graves edited comment on SPARK-23567 at 3/2/18 3:57 PM:

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-03-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387846#comment-16387846 ] Thomas Graves commented on SPARK-16630: --- yes something along these lines is what I

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-03-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387938#comment-16387938 ] Thomas Graves commented on SPARK-22683: --- [~jcuquemelle] do you have time to update

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-03-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387963#comment-16387963 ] Thomas Graves commented on SPARK-22683: --- I left comments on the open PR already, le

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-03-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390232#comment-16390232 ] Thomas Graves commented on SPARK-16630: --- yes yarn tells you the # of nodemanagers. 

[jira] [Commented] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

2018-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416309#comment-16416309 ] Thomas Graves commented on SPARK-22618: --- thanks for fixing this, hitting it now in

[jira] [Commented] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

2018-03-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417466#comment-16417466 ] Thomas Graves commented on SPARK-22618: --- I'll file a separate Jira for it and put u

[jira] [Created] (SPARK-23806) Broadcast. unpersist can cause fatal exception when used with dynamic allocation

2018-03-28 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-23806: - Summary: Broadcast. unpersist can cause fatal exception when used with dynamic allocation Key: SPARK-23806 URL: https://issues.apache.org/jira/browse/SPARK-23806 Pr

[jira] [Updated] (SPARK-23806) Broadcast. unpersist can cause fatal exception when used with dynamic allocation

2018-03-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23806: -- Description: Very similar to https://issues.apache.org/jira/browse/SPARK-22618 . But this coul

[jira] [Created] (SPARK-23850) We should not redact username|user|url from UI by default

2018-04-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-23850: - Summary: We should not redact username|user|url from UI by default Key: SPARK-23850 URL: https://issues.apache.org/jira/browse/SPARK-23850 Project: Spark I

[jira] [Commented] (SPARK-23850) We should not redact username|user|url from UI by default

2018-04-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422965#comment-16422965 ] Thomas Graves commented on SPARK-23850: --- ping [~ash...@gmail.com] [~onursatici] [~L

[jira] [Commented] (SPARK-23039) Fix the bug in alter table set location.

2018-04-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423996#comment-16423996 ] Thomas Graves commented on SPARK-23039: --- seems to be a dup of  # SPARK-23057 > F

[jira] [Resolved] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

2018-04-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23567. --- Resolution: Duplicate > spark.redaction.regex should not include user by default, docs not up

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-04-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427377#comment-16427377 ] Thomas Graves commented on SPARK-16630: --- the problem is that spark.executor.instanc

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-04-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432244#comment-16432244 ] Thomas Graves commented on SPARK-16630: --- yes I think it would make sense as the uni

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-04-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432467#comment-16432467 ] Thomas Graves commented on SPARK-16630: --- sorry I don't follow, the list we get from

[jira] [Created] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-23964: - Summary: why does Spillable wait for 32 elements? Key: SPARK-23964 URL: https://issues.apache.org/jira/browse/SPARK-23964 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Environment: (was: The spillable class has a check: if (elementsRead % 32 == 0 && currentMe

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Description: The spillable class has a check: if (elementsRead % 32 == 0 && currentMemory >= m

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434454#comment-16434454 ] Thomas Graves commented on SPARK-23964: --- [~andrewor14]  [~matei] [~r...@databricks.

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Description: The spillable class has a check in maybeSpill as to when it tries to acquire more

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434479#comment-16434479 ] Thomas Graves commented on SPARK-23964: --- I'm not sure, I'm trying to figure out if

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2018-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440855#comment-16440855 ] Thomas Graves commented on SPARK-15703: --- this Jira is purely making the size of the

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

2018-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444043#comment-16444043 ] Thomas Graves commented on SPARK-23964: --- so far in my testing I haven't seen any pe

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289613#comment-16289613 ] Thomas Graves commented on SPARK-22765: --- why doesn't idle timeout very small < 5 se

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2017-12-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289614#comment-16289614 ] Thomas Graves commented on SPARK-22683: --- So the issue brought up here seems to be r

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2017-12-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289656#comment-16289656 ] Thomas Graves commented on SPARK-22683: --- I am also curious, when you are comparing

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289700#comment-16289700 ] Thomas Graves commented on SPARK-22765: --- ok so its basically they idle timeout duri

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289793#comment-16289793 ] Thomas Graves commented on SPARK-22765: --- ok so before you do anything else I would

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2017-12-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291037#comment-16291037 ] Thomas Graves commented on SPARK-22683: --- Another way to approach this is to have a

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

2017-12-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292569#comment-16292569 ] Thomas Graves commented on SPARK-22465: --- I don't have time at the moment to work on

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292594#comment-16292594 ] Thomas Graves commented on SPARK-22765: --- I'm not sure how mr style and 4-core execu

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

2017-12-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292608#comment-16292608 ] Thomas Graves commented on SPARK-22465: --- Yes I think that makes sense. > Cogroup

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2017-12-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292855#comment-16292855 ] Thomas Graves commented on SPARK-22683: --- Thanks for the clarification, a few of tho

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16296811#comment-16296811 ] Thomas Graves commented on SPARK-22765: --- [~CodingCat] with SPARK-21656 executors sh

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297307#comment-16297307 ] Thomas Graves commented on SPARK-22765: --- yes between stages becomes a problem with

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297335#comment-16297335 ] Thomas Graves commented on SPARK-22765: --- SPARK-21656 and the dynamic allocation sho

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297549#comment-16297549 ] Thomas Graves commented on SPARK-22765: --- with SPARK-21656 does upfront allocation v

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

2017-12-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298589#comment-16298589 ] Thomas Graves commented on SPARK-22765: --- ok so at least based on this one experimen

[jira] [Commented] (SPARK-22577) executor page blacklist status should update with TaskSet level blacklisting

2017-12-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298854#comment-16298854 ] Thomas Graves commented on SPARK-22577: --- Yes I was thinking the stage UI (and perha

[jira] [Commented] (SPARK-22577) executor page blacklist status should update with TaskSet level blacklisting

2017-12-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298874#comment-16298874 ] Thomas Graves commented on SPARK-22577: --- correct (unfortunately don't time), anyone

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

2018-01-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309910#comment-16309910 ] Thomas Graves commented on SPARK-22683: --- [~jcuquemelle] . just to confirm your appl

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2018-01-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331045#comment-16331045 ] Thomas Graves commented on SPARK-20928: --- what is status of this, it looks like subt

[jira] [Commented] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335808#comment-16335808 ] Thomas Graves commented on SPARK-23189: --- [~attilapiros] what do you mean by the las

[jira] [Commented] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336332#comment-16336332 ] Thomas Graves commented on SPARK-23189: --- Personally I use the executors page a lot

[jira] [Commented] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338005#comment-16338005 ] Thomas Graves commented on SPARK-23189: --- for large jobs the specific stage page is

[jira] [Created] (SPARK-23304) Spark SQL coalesce() against hive not working

2018-02-01 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-23304: - Summary: Spark SQL coalesce() against hive not working Key: SPARK-23304 URL: https://issues.apache.org/jira/browse/SPARK-23304 Project: Spark Issue Type: B

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

2018-02-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349212#comment-16349212 ] Thomas Graves commented on SPARK-23304: --- yes there are difference in the # of parti

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

2018-02-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349220#comment-16349220 ] Thomas Graves commented on SPARK-23304: --- If it helps , spark 2.3 # partitions is 31

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

2018-02-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349227#comment-16349227 ] Thomas Graves commented on SPARK-23304: --- Ok, I just realized what you are getting a

[jira] [Updated] (SPARK-23304) Spark SQL coalesce() against hive not working

2018-02-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23304: -- Priority: Major (was: Blocker) > Spark SQL coalesce() against hive not working > -

<    1   2   3   4   5   6   7   8   9   10   >