[jira] [Commented] (SPARK-27816) make TreeNode tag type safe

[ https://issues.apache.org/jira/browse/SPARK-27816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847624#comment-16847624 ] Mark Hamstra commented on SPARK-27816: -- Jira issues with no description are really irritating –

[jira] [Commented] (SPARK-27726) Performance of InMemoryStore suffers under load

[ https://issues.apache.org/jira/browse/SPARK-27726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840636#comment-16840636 ] Mark Hamstra commented on SPARK-27726: -- [~vanzin] > Performance of InMemoryStore suffers under

[jira] [Commented] (SPARK-26502) Get rid of hiveResultString() in QueryExecution

[ https://issues.apache.org/jira/browse/SPARK-26502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732435#comment-16732435 ] Mark Hamstra commented on SPARK-26502: -- Don't lose track of this comment:

[jira] [Updated] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

[ https://issues.apache.org/jira/browse/SPARK-21084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-21084: - Description: One important application of Spark is to support many notebook users with a single

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355943#comment-16355943 ] Mark Hamstra commented on SPARK-22683: -- I agree that setting the config to 1 should be sufficient to

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355895#comment-16355895 ] Mark Hamstra commented on SPARK-22683: -- A concern that I have is that the discussion seems to be

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113605#comment-16113605 ] Mark Hamstra commented on SPARK-21619: -- But part of the point of the split in my half-baked example

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113586#comment-16113586 ] Mark Hamstra commented on SPARK-21619: -- Or you can just enlighten me on how one should design a

[jira] [Comment Edited] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113555#comment-16113555 ] Mark Hamstra edited comment on SPARK-21619 at 8/3/17 10:01 PM: --- Yes, I

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113571#comment-16113571 ] Mark Hamstra commented on SPARK-21619: -- _"Why would you want to execute multiple semantically

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113555#comment-16113555 ] Mark Hamstra commented on SPARK-21619: -- Yes, I absolutely understand that this issue and PR are

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113526#comment-16113526 ] Mark Hamstra commented on SPARK-21619: -- Two reason, mostly: 1) To provide better guarantees that

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113510#comment-16113510 ] Mark Hamstra commented on SPARK-21619: -- Ok, but my point is that if plans are to be canonicalized

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113494#comment-16113494 ] Mark Hamstra commented on SPARK-21619: -- Can you provide a little more context, Reynold, since on its

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830519#comment-15830519 ] Mark Hamstra commented on SPARK-19276: -- Ok, I haven't read your PR closely yet, so I missed that.

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830387#comment-15830387 ] Mark Hamstra commented on SPARK-19276: -- This all makes sense, and the PR is a good effort to fix

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752098#comment-15752098 ] Mark Hamstra commented on SPARK-18886: -- That's a great explanation of the issue, and nice example

[jira] [Commented] (SPARK-17064) Reconsider spark.job.interruptOnCancel

[ https://issues.apache.org/jira/browse/SPARK-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729602#comment-15729602 ] Mark Hamstra commented on SPARK-17064: -- Related JIRA and PR:

[jira] [Commented] (SPARK-17064) Reconsider spark.job.interruptOnCancel

[ https://issues.apache.org/jira/browse/SPARK-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706502#comment-15706502 ] Mark Hamstra commented on SPARK-17064: -- Ignore above PR comment -- wrong JIRA ticket at first >

[jira] [Created] (SPARK-18631) Avoid making data skew worse in ExchangeCoordinator

Mark Hamstra created SPARK-18631: Summary: Avoid making data skew worse in ExchangeCoordinator Key: SPARK-18631 URL: https://issues.apache.org/jira/browse/SPARK-18631 Project: Spark Issue

[jira] [Commented] (SPARK-17911) Scheduler does not need messageScheduler for ResubmitFailedStages

[ https://issues.apache.org/jira/browse/SPARK-17911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583334#comment-15583334 ] Mark Hamstra commented on SPARK-17911: -- I think we're pretty much on the same page when it comes to

[jira] [Created] (SPARK-17769) Some FetchFailure refactoring in the DAGScheduler

Mark Hamstra created SPARK-17769: Summary: Some FetchFailure refactoring in the DAGScheduler Key: SPARK-17769 URL: https://issues.apache.org/jira/browse/SPARK-17769 Project: Spark Issue

[jira] [Updated] (SPARK-17529) On highly skewed data, outer join merges are slow

[ https://issues.apache.org/jira/browse/SPARK-17529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-17529: - Priority: Major (was: Trivial) > On highly skewed data, outer join merges are slow >

[jira] [Updated] (SPARK-17064) Reconsider spark.job.interruptOnCancel

[ https://issues.apache.org/jira/browse/SPARK-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-17064: - Description: There is a frequent need or desire in Spark to cancel already running Tasks. This

[jira] [Commented] (SPARK-17064) Reconsider spark.job.interruptOnCancel

[ https://issues.apache.org/jira/browse/SPARK-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421503#comment-15421503 ] Mark Hamstra commented on SPARK-17064: -- [~kayousterhout] [~r...@databricks.com] [~imranr] >

[jira] [Created] (SPARK-17064) Reconsider spark.job.interruptOnCancel

Mark Hamstra created SPARK-17064: Summary: Reconsider spark.job.interruptOnCancel Key: SPARK-17064 URL: https://issues.apache.org/jira/browse/SPARK-17064 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392517#comment-15392517 ] Mark Hamstra commented on SPARK-16693: -- As much as makes sense and is possible, we should also

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390865#comment-15390865 ] Mark Hamstra commented on SPARK-16693: -- The 2.1.0 release is the very earliest that we can allow

[jira] [Commented] (SPARK-11153) Turns off Parquet filter push-down for string and binary columns

[ https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311252#comment-15311252 ] Mark Hamstra commented on SPARK-11153: -- If I am not mistaken, Parquet 1.8.1 and filter push-down for

[jira] [Commented] (SPARK-15176) Job Scheduling Within Application Suffers from Priority Inversion

[ https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311073#comment-15311073 ] Mark Hamstra commented on SPARK-15176: -- I'm not strongly committed to any API (other than the fact

[jira] [Commented] (SPARK-15176) Job Scheduling Within Application Suffers from Priority Inversion

[ https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304589#comment-15304589 ] Mark Hamstra commented on SPARK-15176: -- It's not an unreasonable use case, and is similar in many

[jira] [Commented] (SPARK-14582) Increase the parallelism for small tables

[ https://issues.apache.org/jira/browse/SPARK-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240389#comment-15240389 ] Mark Hamstra commented on SPARK-14582: -- The total absence of any description in both this JIRA and

[jira] [Comment Edited] (SPARK-9882) Priority-based scheduling for Spark applications

[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234024#comment-15234024 ] Mark Hamstra edited comment on SPARK-9882 at 4/10/16 9:49 AM: -- This isn't a

[jira] [Commented] (SPARK-9882) Priority-based scheduling for Spark applications

[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234024#comment-15234024 ] Mark Hamstra commented on SPARK-9882: - This isn't a very well written JIRA. You are just duplicating

[jira] [Commented] (SPARK-13806) SQL round() produces incorrect results for negative values

[ https://issues.apache.org/jira/browse/SPARK-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205134#comment-15205134 ] Mark Hamstra commented on SPARK-13806: -- Yes, there is the mostly orthogonal question about which

[jira] [Commented] (SPARK-13872) Memory leak in SortMergeOuterJoin

[ https://issues.apache.org/jira/browse/SPARK-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196561#comment-15196561 ] Mark Hamstra commented on SPARK-13872: -- [~joshrosen] > Memory leak in SortMergeOuterJoin >

[jira] [Updated] (SPARK-13806) SQL round() produces incorrect results for negative values

[ https://issues.apache.org/jira/browse/SPARK-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-13806: - Description: Round in catalyst/expressions/mathExpressions.scala appears to be untested with

[jira] [Updated] (SPARK-13806) SQL round() produces incorrect results for negative values

[ https://issues.apache.org/jira/browse/SPARK-13806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-13806: - Description: Round in catalyst/expressions/mathExpressions.scala appears to be untested with

[jira] [Created] (SPARK-13806) SQL round() produces incorrect results for negative values

Mark Hamstra created SPARK-13806: Summary: SQL round() produces incorrect results for negative values Key: SPARK-13806 URL: https://issues.apache.org/jira/browse/SPARK-13806 Project: Spark

[jira] [Updated] (SPARK-11838) Spark SQL query fragment RDD reuse across queries

[ https://issues.apache.org/jira/browse/SPARK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-11838: - Summary: Spark SQL query fragment RDD reuse across queries (was: Spark SQL query fragment RDD

[jira] [Comment Edited] (SPARK-13756) Reuse Query Fragments

[ https://issues.apache.org/jira/browse/SPARK-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185991#comment-15185991 ] Mark Hamstra edited comment on SPARK-13756 at 3/8/16 10:42 PM: --- Collecting

[jira] [Updated] (SPARK-13756) Reuse Query Fragments

[ https://issues.apache.org/jira/browse/SPARK-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-13756: - Description: Query fragments that have been materialized in RDDs can and should be reused either

[jira] [Commented] (SPARK-13756) Reuse Query Fragments

[ https://issues.apache.org/jira/browse/SPARK-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185991#comment-15185991 ] Mark Hamstra commented on SPARK-13756: -- Fragment reuse across queries > Reuse Query Fragments >

[jira] [Comment Edited] (SPARK-13523) Reuse the exchanges in a query

[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra edited comment on SPARK-13523 at 3/8/16 10:36 PM: --- Yes that is

[jira] [Comment Edited] (SPARK-13523) Reuse the exchanges in a query

[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra edited comment on SPARK-13523 at 3/8/16 10:36 PM: --- Yes that is

[jira] [Created] (SPARK-13756) Reuse Query Fragments

Mark Hamstra created SPARK-13756: Summary: Reuse Query Fragments Key: SPARK-13756 URL: https://issues.apache.org/jira/browse/SPARK-13756 Project: Spark Issue Type: Umbrella

[jira] [Commented] (SPARK-13523) Reuse the exchanges in a query

[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185967#comment-15185967 ] Mark Hamstra commented on SPARK-13523: -- Yes that is a good point. But they are closely enough

[jira] [Resolved] (SPARK-13523) Reuse the exchanges in a query

[ https://issues.apache.org/jira/browse/SPARK-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra resolved SPARK-13523. -- Resolution: Duplicate > Reuse the exchanges in a query > -- > >

[jira] [Created] (SPARK-13087) Grouping by a complex expression may lead to incorrect AttributeReferences in aggregations

Mark Hamstra created SPARK-13087: Summary: Grouping by a complex expression may lead to incorrect AttributeReferences in aggregations Key: SPARK-13087 URL: https://issues.apache.org/jira/browse/SPARK-13087

[jira] [Commented] (SPARK-8279) udf_round_3 test fails

[ https://issues.apache.org/jira/browse/SPARK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117924#comment-15117924 ] Mark Hamstra commented on SPARK-8279: - It's a bit of an under-defined mess, actually. I don't have a

[jira] [Commented] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

[ https://issues.apache.org/jira/browse/SPARK-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106820#comment-15106820 ] Mark Hamstra commented on SPARK-12485: -- Actually, Sean, I'd argue that they are not the same and

[jira] [Commented] (SPARK-12485) Rename "dynamic allocation" to "elastic scaling"

[ https://issues.apache.org/jira/browse/SPARK-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106966#comment-15106966 ] Mark Hamstra commented on SPARK-12485: -- Right, I am saying that "dynamic allocation" is the right

[jira] [Commented] (SPARK-8279) udf_round_3 test fails

[ https://issues.apache.org/jira/browse/SPARK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090098#comment-15090098 ] Mark Hamstra commented on SPARK-8279: - Why is `round` in Spark SQL using HALF_UP instead of the

[jira] [Commented] (SPARK-6416) RDD.fold() requires the operator to be commutative

[ https://issues.apache.org/jira/browse/SPARK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076704#comment-15076704 ] Mark Hamstra commented on SPARK-6416: - I still don't see RDD#fold as being out of bounds with what

[jira] [Commented] (SPARK-6416) RDD.fold() requires the operator to be commutative

[ https://issues.apache.org/jira/browse/SPARK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076367#comment-15076367 ] Mark Hamstra commented on SPARK-6416: - I don't see any reason to change the API wrt `fold`. With

[jira] [Updated] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-12258: - Component/s: SQL > Hive Timestamp UDF is binded with '1969-12-31 15:59:59.99' for null value

[jira] [Commented] (SPARK-11838) Spark SQL query fragment RDD reuse

[ https://issues.apache.org/jira/browse/SPARK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012724#comment-15012724 ] Mark Hamstra commented on SPARK-11838: -- One significant difference between this and CacheManager is

[jira] [Commented] (SPARK-11153) Turns off Parquet filter push-down for string and binary columns

[ https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005542#comment-15005542 ] Mark Hamstra commented on SPARK-11153: -- Thanks. > Turns off Parquet filter push-down for string and

[jira] [Commented] (SPARK-11153) Turns off Parquet filter push-down for string and binary columns

[ https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004944#comment-15004944 ] Mark Hamstra commented on SPARK-11153: -- Is there a reason why parquet.version hasn't been pushed up

[jira] [Commented] (SPARK-11326) Split networking in standalone mode

[ https://issues.apache.org/jira/browse/SPARK-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992282#comment-14992282 ] Mark Hamstra commented on SPARK-11326: -- Can someone clarify for me just what is motivating this JIRA

[jira] [Commented] (SPARK-11326) Split networking in standalone mode

[ https://issues.apache.org/jira/browse/SPARK-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992478#comment-14992478 ] Mark Hamstra commented on SPARK-11326: -- [~rspitzer] A focus on reaching feature parity is definitely

[jira] [Commented] (SPARK-11539) Debian packaging

[ https://issues.apache.org/jira/browse/SPARK-11539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992601#comment-14992601 ] Mark Hamstra commented on SPARK-11539: -- I agree with [~srowen]. See

[jira] [Commented] (SPARK-11326) Split networking in standalone mode

[ https://issues.apache.org/jira/browse/SPARK-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992468#comment-14992468 ] Mark Hamstra commented on SPARK-11326: -- bq. I do believe that there are quite a lot of people who

[jira] [Commented] (SPARK-10723) Add RDD.reduceOption method

[ https://issues.apache.org/jira/browse/SPARK-10723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900763#comment-14900763 ] Mark Hamstra commented on SPARK-10723: -- Either check `isEmpty` or use `fold` to handle empty RDDs.

[jira] [Assigned] (SPARK-10707) Set operation output columns may have incorrect nullability

[ https://issues.apache.org/jira/browse/SPARK-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra reassigned SPARK-10707: Assignee: Mark Hamstra > Set operation output columns may have incorrect nullability >

[jira] [Created] (SPARK-10666) Use properties from ActiveJob associated with a Stage

Mark Hamstra created SPARK-10666: Summary: Use properties from ActiveJob associated with a Stage Key: SPARK-10666 URL: https://issues.apache.org/jira/browse/SPARK-10666 Project: Spark Issue

[jira] [Commented] (SPARK-6880) Spark Shutdowns with NoSuchElementException when running parallel collect on cachedRDD

[ https://issues.apache.org/jira/browse/SPARK-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803261#comment-14803261 ] Mark Hamstra commented on SPARK-6880: - see SPARK-10666 > Spark Shutdowns with NoSuchElementException

[jira] [Commented] (SPARK-6416) RDD.fold() requires the operator to be commutative

[ https://issues.apache.org/jira/browse/SPARK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554871#comment-14554871 ] Mark Hamstra commented on SPARK-6416: - Why remove it? It's very useful when used

[jira] [Commented] (SPARK-7750) Rename json endpoints to api endpoints

[ https://issues.apache.org/jira/browse/SPARK-7750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551718#comment-14551718 ] Mark Hamstra commented on SPARK-7750: - Including `@Produces` annotations is also

[jira] [Commented] (SPARK-6880) Spark Shutdowns with NoSuchElementException when running parallel collect on cachedRDD

[ https://issues.apache.org/jira/browse/SPARK-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551647#comment-14551647 ] Mark Hamstra commented on SPARK-6880: - This fix should also be applied as far back as

[jira] [Commented] (SPARK-839) Bug in how failed executors are removed by ID from standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311734#comment-14311734 ] Mark Hamstra commented on SPARK-839: Fixed long ago. Bug in how failed executors are

[jira] [Closed] (SPARK-839) Bug in how failed executors are removed by ID from standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra closed SPARK-839. -- Resolution: Fixed Bug in how failed executors are removed by ID from standalone cluster

[jira] [Commented] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230910#comment-14230910 ] Mark Hamstra commented on SPARK-4498: - I'd argue against reverting 2425 on the grounds

[jira] [Commented] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228947#comment-14228947 ] Mark Hamstra commented on SPARK-4498: - On a quick look-through, your analysis looks

[jira] [Closed] (SPARK-4473) [Core] StageInfo should have ActiveJob's group ID as a field

[ https://issues.apache.org/jira/browse/SPARK-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra closed SPARK-4473. --- Resolution: Duplicate [Core] StageInfo should have ActiveJob's group ID as a field

[jira] [Commented] (SPARK-4473) [Core] StageInfo should have ActiveJob's group ID as a field

[ https://issues.apache.org/jira/browse/SPARK-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218057#comment-14218057 ] Mark Hamstra commented on SPARK-4473: - This is already covered by

[jira] [Created] (SPARK-4436) Debian packaging misses datanucleus jars

Mark Hamstra created SPARK-4436: --- Summary: Debian packaging misses datanucleus jars Key: SPARK-4436 URL: https://issues.apache.org/jira/browse/SPARK-4436 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-4436) Debian packaging misses datanucleus jars

[ https://issues.apache.org/jira/browse/SPARK-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-4436: Description: If Spark is built with Hive support (i.e. -Phive), then the necessary datanucleus

[jira] [Commented] (SPARK-4428) Use ${scala.binary.version} property for artifactId.

[ https://issues.apache.org/jira/browse/SPARK-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213664#comment-14213664 ] Mark Hamstra commented on SPARK-4428: - This is not a bug, nor is it a major issue, nor

[jira] [Resolved] (SPARK-4428) Use ${scala.binary.version} property for artifactId.

[ https://issues.apache.org/jira/browse/SPARK-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra resolved SPARK-4428. - Resolution: Won't Fix Use ${scala.binary.version} property for artifactId.

[jira] [Commented] (SPARK-2321) Design a proper progress reporting event listener API

[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143814#comment-14143814 ] Mark Hamstra commented on SPARK-2321: - Which would be kind of the opposite half of the

[jira] [Updated] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-1021: Assignee: Erik Erlandson (was: Mark Hamstra) sortByKey() launches a cluster job when it shouldn't

[jira] [Commented] (SPARK-3289) Avoid job failures due to rescheduling of failing tasks on buggy machines

[ https://issues.apache.org/jira/browse/SPARK-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114611#comment-14114611 ] Mark Hamstra commented on SPARK-3289: - https://github.com/apache/spark/pull/1360

[jira] [Updated] (SPARK-2714) DAGScheduler logs jobid when runJob finishes

[ https://issues.apache.org/jira/browse/SPARK-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-2714: Issue Type: Improvement (was: Documentation) DAGScheduler logs jobid when runJob finishes

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076778#comment-14076778 ] Mark Hamstra commented on SPARK-1860: - I don't think that there is much in the way of

[jira] [Commented] (SPARK-1812) Support cross-building with Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075176#comment-14075176 ] Mark Hamstra commented on SPARK-1812: - FWIW scalatest can be pushed to 2.2.0 without

[jira] [Commented] (SPARK-2614) Add the spark-examples-xxx-.jar to the Debian package created by assembly/pom.xml (e.g. -Pdeb)

[ https://issues.apache.org/jira/browse/SPARK-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070908#comment-14070908 ] Mark Hamstra commented on SPARK-2614: - It's also common for installers/admins to not

[jira] [Commented] (SPARK-2568) RangePartitioner should go through the data only once

[ https://issues.apache.org/jira/browse/SPARK-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066553#comment-14066553 ] Mark Hamstra commented on SPARK-2568: - Sure, if they can be cleanly separated -- but

[jira] [Resolved] (SPARK-2158) FileAppenderSuite is not cleaning up after itself

[ https://issues.apache.org/jira/browse/SPARK-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra resolved SPARK-2158. - Resolution: Fixed FileAppenderSuite is not cleaning up after itself

[jira] [Commented] (SPARK-2158) FileAppenderSuite is not cleaning up after itself

[ https://issues.apache.org/jira/browse/SPARK-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060201#comment-14060201 ] Mark Hamstra commented on SPARK-2158: - This is fixed at 4cb33a83e0 from

[jira] [Created] (SPARK-2424) ApplicationState.MAX_NUM_RETRY should be configurable

Mark Hamstra created SPARK-2424: --- Summary: ApplicationState.MAX_NUM_RETRY should be configurable Key: SPARK-2424 URL: https://issues.apache.org/jira/browse/SPARK-2424 Project: Spark Issue

[jira] [Created] (SPARK-2425) Standalone Master is too aggressive in removing Applications

Mark Hamstra created SPARK-2425: --- Summary: Standalone Master is too aggressive in removing Applications Key: SPARK-2425 URL: https://issues.apache.org/jira/browse/SPARK-2425 Project: Spark

[jira] [Commented] (SPARK-2198) Partition the scala build file so that it is easier to maintain

[ https://issues.apache.org/jira/browse/SPARK-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14037431#comment-14037431 ] Mark Hamstra commented on SPARK-2198: - While this is an admirable goal, I'm afraid

[jira] [Updated] (SPARK-2126) Move MapOutputTracker behind ShuffleManager interface

[ https://issues.apache.org/jira/browse/SPARK-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-2126: Assignee: Nan Zhu Move MapOutputTracker behind ShuffleManager interface

[jira] [Commented] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031257#comment-14031257 ] Mark Hamstra commented on SPARK-1201: - What causes this to not be fixable within the

[jira] [Commented] (SPARK-1201) Do not materialize partitions whenever possible in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031426#comment-14031426 ] Mark Hamstra commented on SPARK-1201: - Okay, but my question is really whether

[jira] [Commented] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017776#comment-14017776 ] Mark Hamstra commented on SPARK-2019: - Please don't leave the Affects Version/s

[jira] [Updated] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-2019: Affects Version/s: 0.9.1 Spark workers die/disappear when job fails for nearly any reason

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015671#comment-14015671 ] Mark Hamstra commented on SPARK-983: Is that code visible someplace? In broad outline,

[jira] [Updated] (SPARK-1973) Add randomSplit to JavaRDD (with tests, and tidy Java tests)

[ https://issues.apache.org/jira/browse/SPARK-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-1973: Fix Version/s: 1.1.0 Add randomSplit to JavaRDD (with tests, and tidy Java tests)

  1   2   >