[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-05-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487901#comment-16487901 ] Imran Rashid commented on SPARK-23206: -- [~felixcheung] can you give an example of th

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-22 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484069#comment-16484069 ] Imran Rashid commented on SPARK-6235: - Would be nice to find a better home for this, b

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482901#comment-16482901 ] Imran Rashid commented on SPARK-6235: - [~tgraves] WAL -- write-ahead-log for receiver-

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482699#comment-16482699 ] Imran Rashid commented on SPARK-6235: - Linked a [design doc|https://docs.google.com/d

[jira] [Updated] (SPARK-24309) AsyncEventQueue should handle an interrupt from a Listener

2018-05-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-24309: - Target Version/s: 2.3.1 > AsyncEventQueue should handle an interrupt from a Listener > --

[jira] [Updated] (SPARK-24309) AsyncEventQueue should handle an interrupt from a Listener

2018-05-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-24309: - Priority: Blocker (was: Major) > AsyncEventQueue should handle an interrupt from a Listener > --

[jira] [Created] (SPARK-24309) AsyncEventQueue should handle an interrupt from a Listener

2018-05-17 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24309: Summary: AsyncEventQueue should handle an interrupt from a Listener Key: SPARK-24309 URL: https://issues.apache.org/jira/browse/SPARK-24309 Project: Spark Is

[jira] [Commented] (SPARK-24307) Support sending messages over 2GB from memory

2018-05-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479439#comment-16479439 ] Imran Rashid commented on SPARK-24307: -- I have a really hacky version of this now, I

[jira] [Created] (SPARK-24307) Support sending messages over 2GB from memory

2018-05-17 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24307: Summary: Support sending messages over 2GB from memory Key: SPARK-24307 URL: https://issues.apache.org/jira/browse/SPARK-24307 Project: Spark Issue Type: Sub

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477986#comment-16477986 ] Imran Rashid commented on SPARK-6235: - derp, I missed a pretty basic case -- if you ca

[jira] [Resolved] (SPARK-5928) Remote Shuffle Blocks cannot be more than 2 GB

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-5928. - Resolution: Duplicate this was solved in SPARK-19659, as long as you set spark.maxRemoteBlockSize

[jira] [Updated] (SPARK-6237) Support uploading blocks > 2GB as a stream

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-6237: Summary: Support uploading blocks > 2GB as a stream (was: Support network transfer for blocks large

[jira] [Resolved] (SPARK-17184) Replace ByteBuf with InputStream

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-17184. -- Resolution: Incomplete I'm going to close this as there isn't really enough detail here to say

[jira] [Resolved] (SPARK-17082) Replace ByteBuffer with ChunkedByteBuffer

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-17082. -- Resolution: Incomplete I'm going to close this as there isn't really enough detail here to say

[jira] [Created] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

2018-05-16 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24297: Summary: Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB Key: SPARK-24297 URL: https://issues.apache.org/jira/browse/SPARK-24297 Project: Spar

[jira] [Commented] (SPARK-6236) Support caching blocks larger than 2G

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477857#comment-16477857 ] Imran Rashid commented on SPARK-6236: - I believe this has actually been solved by othe

[jira] [Created] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-16 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24296: Summary: Support replicating blocks larger than 2 GB Key: SPARK-24296 URL: https://issues.apache.org/jira/browse/SPARK-24296 Project: Spark Issue Type: Sub-t

[jira] [Commented] (SPARK-6237) Support network transfer for blocks larger than 2G

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477848#comment-16477848 ] Imran Rashid commented on SPARK-6237: - The original task here ("Support network transf

[jira] [Resolved] (SPARK-6238) Support shuffle where individual blocks might be > 2G

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6238. - Resolution: Duplicate Assignee: jin xing This was solved by [~jinxing6...@126.com] in SPARK-

[jira] [Resolved] (SPARK-6190) create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6190. - Resolution: Duplicate Assignee: Josh Rosen (was: Imran Rashid) This was done by [~joshrosen

[jira] [Commented] (SPARK-6235) Address various 2G limits

2018-05-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477827#comment-16477827 ] Imran Rashid commented on SPARK-6235: - I've been testing the current state of the 2GB

[jira] [Updated] (SPARK-24274) Job UI should make stage dependencies clear in a complex DAG

2018-05-14 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-24274: - Attachment: q95_job17.tgz > Job UI should make stage dependencies clear in a complex DAG > --

[jira] [Created] (SPARK-24274) Job UI should make stage dependencies clear in a complex DAG

2018-05-14 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24274: Summary: Job UI should make stage dependencies clear in a complex DAG Key: SPARK-24274 URL: https://issues.apache.org/jira/browse/SPARK-24274 Project: Spark

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

2018-05-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469335#comment-16469335 ] Imran Rashid edited comment on SPARK-23206 at 5/9/18 7:16 PM: -

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-05-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469335#comment-16469335 ] Imran Rashid commented on SPARK-23206: -- Hi, I think getting together to discuss the

[jira] [Commented] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-05-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467570#comment-16467570 ] Imran Rashid commented on SPARK-23894: -- After discussion in related PRs, SPARK-22938

[jira] [Resolved] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-05-03 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23433. -- Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 2.2.2

[jira] [Assigned] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-05-03 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23433: Assignee: Imran Rashid > java.lang.IllegalStateException: more than one active taskSet for

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-03 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462540#comment-16462540 ] Imran Rashid commented on SPARK-24135: -- Honestly I don't understand the failure mode

[jira] [Commented] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-04-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457003#comment-16457003 ] Imran Rashid commented on SPARK-23894: -- I believe this issue has existed since SPARK

[jira] [Commented] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-04-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456969#comment-16456969 ] Imran Rashid commented on SPARK-23894: -- I think I understand what is happening here,

[jira] [Commented] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-04-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456919#comment-16456919 ] Imran Rashid commented on SPARK-23894: -- One thing I've noticed from looking at more

[jira] [Commented] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2018-04-25 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452369#comment-16452369 ] Imran Rashid commented on SPARK-20087: -- Sound good to me, I'm in favor of the change

[jira] [Assigned] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

2018-04-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23888: Assignee: wuyi > speculative task should not run on a given host where another attempt is

[jira] [Resolved] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

2018-04-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23888. -- Resolution: Fixed Fix Version/s: (was: 2.3.0) 2.4.0 Issue resolve

[jira] [Resolved] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-24021. -- Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull r

[jira] [Assigned] (SPARK-24021) Fix bug in BlacklistTracker's updateBlacklistForFetchFailure

2018-04-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-24021: Assignee: wuyi > Fix bug in BlacklistTracker's updateBlacklistForFetchFailure > --

[jira] [Created] (SPARK-24016) Yarn does not update node blacklist in static allocation

2018-04-18 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24016: Summary: Yarn does not update node blacklist in static allocation Key: SPARK-24016 URL: https://issues.apache.org/jira/browse/SPARK-24016 Project: Spark Issu

[jira] [Updated] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23948: - Fix Version/s: 2.3.1 > Trigger mapstage's job listener in submitMissingTasks > --

[jira] [Assigned] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23948: Assignee: jin xing > Trigger mapstage's job listener in submitMissingTasks > -

[jira] [Resolved] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23948. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21019 [https://git

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440334#comment-16440334 ] Imran Rashid commented on SPARK-23206: -- thanks, shared doc works for me now! > Addi

[jira] [Updated] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23948: - Component/s: Scheduler > Trigger mapstage's job listener in submitMissingTasks >

[jira] [Assigned] (SPARK-22941) Allow SparkSubmit to throw exceptions instead of exiting / printing errors.

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-22941: Assignee: Marcelo Vanzin > Allow SparkSubmit to throw exceptions instead of exiting / prin

[jira] [Resolved] (SPARK-22941) Allow SparkSubmit to throw exceptions instead of exiting / printing errors.

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-22941. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20925 [https://git

[jira] [Created] (SPARK-23962) Flaky tests from SQLMetricsTestUtils.currentExecutionIds

2018-04-11 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23962: Summary: Flaky tests from SQLMetricsTestUtils.currentExecutionIds Key: SPARK-23962 URL: https://issues.apache.org/jira/browse/SPARK-23962 Project: Spark Issu

[jira] [Updated] (SPARK-23962) Flaky tests from SQLMetricsTestUtils.currentExecutionIds

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23962: - Attachment: unit-tests.log > Flaky tests from SQLMetricsTestUtils.currentExecutionIds > -

[jira] [Resolved] (SPARK-6951) History server slow startup if the event log directory is large

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6951. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20952 [https://github

[jira] [Assigned] (SPARK-6951) History server slow startup if the event log directory is large

2018-04-11 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-6951: --- Assignee: Marcelo Vanzin > History server slow startup if the event log directory is large >

[jira] [Updated] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

2018-04-10 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23888: - Labels: speculation (was: ) > speculative task should not run on a given host where another atte

[jira] [Updated] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

2018-04-10 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23888: - Component/s: Scheduler > speculative task should not run on a given host where another attempt is

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430836#comment-16430836 ] Imran Rashid commented on SPARK-23206: -- ah of course, that makes sense -- quantiles

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430616#comment-16430616 ] Imran Rashid commented on SPARK-23206: -- I have a question about this part of the des

[jira] [Updated] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-04-07 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23894: - Attachment: unit-tests.log > Flaky Test: BucketedWriteWithoutHiveSupportSuite >

[jira] [Created] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite

2018-04-07 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23894: Summary: Flaky Test: BucketedWriteWithoutHiveSupportSuite Key: SPARK-23894 URL: https://issues.apache.org/jira/browse/SPARK-23894 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23816) FetchFailedException when killing speculative task

2018-04-06 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23816: - Labels: speculation (was: ) > FetchFailedException when killing speculative task > -

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-04-03 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424906#comment-16424906 ] Imran Rashid commented on SPARK-16630: -- Be sure to look at the discussion on this PR

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

2018-04-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422588#comment-16422588 ] Imran Rashid commented on SPARK-19276: -- Oh thanks for pointing that out [~xchen12138

[jira] [Resolved] (SPARK-14044) Allow configuration of DynamicPartitionWriterContainer#writeRows to bypass sort step

2018-03-30 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-14044. -- Resolution: Duplicate I'm resolving this as a duplicate of SPARK-19563 -- please re-open if I'm

[jira] [Commented] (SPARK-21834) Incorrect executor request in case of dynamic allocation

2018-03-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411897#comment-16411897 ] Imran Rashid commented on SPARK-21834: -- SPARK-23365 is basically a duplicate of this

[jira] [Commented] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-03-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16411894#comment-16411894 ] Imran Rashid commented on SPARK-23365: -- This is mostly a duplicate of SPARK-21834,

[jira] [Updated] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-03-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23365: - Description: Dynamic Allocation can lead to a spark app getting stuck with 0 executors requested

[jira] [Updated] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-03-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23365: - Description: Dynamic Allocation can lead to a spark app getting stuck with 0 executors requested

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-03-07 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390159#comment-16390159 ] Imran Rashid commented on SPARK-16630: -- I'd also take {{spark.yarn.max.executor.fail

[jira] [Commented] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-03-06 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387759#comment-16387759 ] Imran Rashid commented on SPARK-23433: -- sorry it has taken me some time to get to th

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

2018-02-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375005#comment-16375005 ] Imran Rashid commented on SPARK-23485: -- {quote} I think this is because the general

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

2018-02-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374748#comment-16374748 ] Imran Rashid commented on SPARK-23485: -- ok the missing jar was a bad example on kube

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

2018-02-23 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374599#comment-16374599 ] Imran Rashid commented on SPARK-23485: -- Yeah I don't think its safe to assume that i

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

2018-02-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372134#comment-16372134 ] Imran Rashid commented on SPARK-23485: -- Also related to SPARK-16630 ... if that is s

[jira] [Created] (SPARK-23485) Kubernetes should support node blacklist

2018-02-21 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23485: Summary: Kubernetes should support node blacklist Key: SPARK-23485 URL: https://issues.apache.org/jira/browse/SPARK-23485 Project: Spark Issue Type: New Feat

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-20 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23053: - Fix Version/s: 2.1.3 > taskBinarySerialization and task partitions calculate in > DagScheduler.s

[jira] [Commented] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-02-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367666#comment-16367666 ] Imran Rashid commented on SPARK-23433: -- actually, I realized its more general than j

[jira] [Commented] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-02-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367664#comment-16367664 ] Imran Rashid commented on SPARK-23433: -- yes I think you are right [~zsxwing]. Since

[jira] [Updated] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23413: - Affects Version/s: (was: 2.4.0) > Sorting tasks by Host / Executor ID on the Stage page does

[jira] [Commented] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366199#comment-16366199 ] Imran Rashid commented on SPARK-23413: -- This was fixed by https://github.com/apache/

[jira] [Resolved] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23413. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20623 [https://git

[jira] [Assigned] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23413: Assignee: Attila Zsolt Piros > Sorting tasks by Host / Executor ID on the Stage page does

[jira] [Resolved] (SPARK-23235) Add executor Threaddump to api

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23235. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20474 [https://git

[jira] [Assigned] (SPARK-23235) Add executor Threaddump to api

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23235: Assignee: Attila Zsolt Piros > Add executor Threaddump to api > --

[jira] [Resolved] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23053. -- Resolution: Fixed > taskBinarySerialization and task partitions calculate in > DagScheduler.su

[jira] [Assigned] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23053: Assignee: huangtengfei > taskBinarySerialization and task partitions calculate in > DagSc

[jira] [Commented] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362556#comment-16362556 ] Imran Rashid commented on SPARK-23053: -- Fixed by https://github.com/apache/spark/pul

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23053: - Fix Version/s: 2.4.0 2.3.1 2.2.2 > taskBinarySerialization

[jira] [Assigned] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23189: Assignee: Attila Zsolt Piros > reflect stage level blacklisting on executor tab > ---

[jira] [Resolved] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23189. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20408 [https://git

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2018-02-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358612#comment-16358612 ] Imran Rashid commented on SPARK-19870: -- to be honest, I'm not really sure what I'm l

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357953#comment-16357953 ] Imran Rashid commented on SPARK-23206: -- +1 on all the ideas discussed here so far.

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357949#comment-16357949 ] Imran Rashid commented on SPARK-23235: -- [~jerryshao] thanks for pointing me at SPARK

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357930#comment-16357930 ] Imran Rashid commented on SPARK-19870: -- [~eyalfa] I see that warning from every task

[jira] [Created] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-02-08 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23365: Summary: DynamicAllocation with failure in straggler task can lead to a hung spark job Key: SPARK-23365 URL: https://issues.apache.org/jira/browse/SPARK-23365 Project

[jira] [Commented] (SPARK-23139) Read eventLog file with mixed encodings

2018-02-08 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356989#comment-16356989 ] Imran Rashid commented on SPARK-23139: -- I think some confusion may come from the jir

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2018-02-07 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355982#comment-16355982 ] Imran Rashid commented on SPARK-19870: -- [~eyalfa] any chance you can share those exe

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2018-02-06 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354680#comment-16354680 ] Imran Rashid commented on SPARK-19870: -- [~eyalfa] my recollection is a bit rusty, bu

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-05 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353229#comment-16353229 ] Imran Rashid commented on SPARK-23308: -- well I think the complaint is that you end u

[jira] [Commented] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2018-02-05 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352750#comment-16352750 ] Imran Rashid commented on SPARK-20087: -- cc [~holden.ka...@gmail.com] I think this m

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-05 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352744#comment-16352744 ] Imran Rashid commented on SPARK-23308: -- I think the problem is that its really trick

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23053: - Component/s: Scheduler > taskBinarySerialization and task partitions calculate in > DagScheduler

[jira] [Commented] (SPARK-23139) Read eventLog file with mixed encodings

2018-02-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350933#comment-16350933 ] Imran Rashid commented on SPARK-23139: -- Apologies if this is a really silly question

[jira] [Assigned] (SPARK-23253) Only write shuffle temporary index file when there is not an existing one

2018-02-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23253: Assignee: Kent Yao > Only write shuffle temporary index file when there is not an existing

[jira] [Resolved] (SPARK-23253) Only write shuffle temporary index file when there is not an existing one

2018-02-02 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23253. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20422 [https://git

<    1   2   3   4   5   6   7   8   9   10   >