[jira] [Commented] (SPARK-20683) Make table uncache chaining optional

2017-05-19 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018008#comment-16018008 ] Andrew Ash commented on SPARK-20683: Thanks for that diff [~shea.parkes] -- we're planning on trying

[jira] [Created] (SPARK-20769) Incorrect documentation for using Jupyter notebook

2017-05-16 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-20769: -- Summary: Incorrect documentation for using Jupyter notebook Key: SPARK-20769 URL: https://issues.apache.org/jira/browse/SPARK-20769 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-05-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012011#comment-16012011 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Eagle cluster manager

[jira] [Commented] (SPARK-20429) [GRAPHX] Strange results for personalized pagerank if node is involved in a cycle

2017-05-01 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991099#comment-15991099 ] Andrew Ray commented on SPARK-20429: Can you retest your example with Spark 2.2/master. SPARK-18847

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979630#comment-15979630 ] Andrew Ash commented on SPARK-20433: It's unclear if Spark is affected, I wanted to open this ticket

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Major (was: Blocker) > Security issue with jackson-databind >

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Critical (was: Major) > Security issue with jackson-databind >

[jira] [Created] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-20433: -- Summary: Security issue with jackson-databind Key: SPARK-20433 URL: https://issues.apache.org/jira/browse/SPARK-20433 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973763#comment-15973763 ] Andrew Ash commented on SPARK-20364: Thanks for the investigation [~hyukjin.kwon]! The proposal you

[jira] [Comment Edited] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

2017-04-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966782#comment-15966782 ] Andrew Ash edited comment on SPARK-1809 at 4/12/17 11:00 PM: - I'm not using

[jira] [Closed] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

2017-04-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-1809. - Resolution: Unresolved Not using Mesos anymore, so closing > Mesos backend doesn't respect

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2017-04-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961170#comment-15961170 ] Andrew Ash commented on SPARK-20144: This is a regression from 1.6 to the 2.x line. [~marmbrus]

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

2017-03-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939072#comment-15939072 ] Andrew Ash commented on SPARK-19372: I've seen this as well on parquet files. > Code generation for

[jira] [Resolved] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

2017-03-23 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray resolved SPARK-19136. Resolution: Not A Bug > Aggregator with case class as output type fails with ClassCastException >

[jira] [Updated] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-03-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19528: --- Description: when dynamic allocation is enabled, the external shuffle service is used for maintain

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda environment

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda environment

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929591#comment-15929591 ] Andrew Ash commented on SPARK-18278: As an update on this ticket: For those not already aware, work

[jira] [Comment Edited] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-03-09 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903778#comment-15903778 ] Andrew Milkowski edited comment on SPARK-19364 at 3/9/17 8:17 PM: --

[jira] [Commented] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-03-09 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903778#comment-15903778 ] Andrew Milkowski commented on SPARK-19364: -- thanks @Takeshi Yamamuro , will try to see if I can

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-03-02 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893222#comment-15893222 ] Andrew Ash commented on SPARK-18113: We discovered another bug related to committing that causes task

[jira] [Commented] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0

2017-03-02 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893074#comment-15893074 ] Andrew Otto commented on SPARK-1693: We just upgraded to CDH 5.10, which has Spark 1.6.0, Hadoop

[jira] [Comment Edited] (SPARK-1693) Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0

2017-03-02 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893074#comment-15893074 ] Andrew Otto edited comment on SPARK-1693 at 3/2/17 9:42 PM: We just upgraded

[jira] [Commented] (SPARK-7354) Flaky test: o.a.s.deploy.SparkSubmitSuite --jars

2017-02-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881593#comment-15881593 ] Andrew Ash commented on SPARK-7354: --- We saw a flake for this test in the k8s repo's Travis builds too:

[jira] [Commented] (SPARK-19326) Speculated task attempts do not get launched in few scenarios

2017-02-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869244#comment-15869244 ] Andrew Or commented on SPARK-19326: --- I would say it's a bad idea to make ExecutorAllocationManager talk

[jira] [Commented] (SPARK-19326) Speculated task attempts do not get launched in few scenarios

2017-02-15 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869020#comment-15869020 ] Andrew Or commented on SPARK-19326: --- Sorry for slipping on this. When I was implementing the feature

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865778#comment-15865778 ] Andrew Ash commented on SPARK-18113: [~xukun] the scenario you describe should be accommodated by the

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865536#comment-15865536 ] Andrew Ash commented on SPARK-18113: Thanks for the updates you both. I've been working with a

[jira] [Commented] (SPARK-19493) Remove Java 7 support

2017-02-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861576#comment-15861576 ] Andrew Ash commented on SPARK-19493: +1 -- we're removing Java 7 compatibility from core internal

[jira] [Updated] (SPARK-19522) --executor-memory flag doesn't work in local-cluster mode

2017-02-08 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-19522: -- Description: {code} bin/spark-shell --master local-cluster[2,1,2048] {code} doesn't do what you think

[jira] [Created] (SPARK-19522) --executor-memory flag doesn't work in local-cluster mode

2017-02-08 Thread Andrew Or (JIRA)
Andrew Or created SPARK-19522: - Summary: --executor-memory flag doesn't work in local-cluster mode Key: SPARK-19522 URL: https://issues.apache.org/jira/browse/SPARK-19522 Project: Spark Issue

[jira] [Updated] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Priority: Blocker (was: Major) > Stream Blocks in Storage Persists Forever when Kinesis

[jira] [Updated] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Description: -- update --- we found that below situation occurs when we encounter

[jira] [Updated] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Summary: Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled

[jira] [Updated] (SPARK-19364) Some Stream Blocks in Storage Persists Forever

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Description: -- update --- we found that below situation occurs when we encounter

[jira] [Updated] (SPARK-19364) Some Stream Blocks in Storage Persists Forever

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Description: -- update --- we found that below situation occurs when we encounter

[jira] [Updated] (SPARK-19364) Some Stream Blocks in Storage Persists Forever

2017-01-26 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Description: *** update *** we found that below situation occurs when we encounter

[jira] [Updated] (SPARK-19364) Some Stream Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Summary: Some Stream Blocks in Storage Persists Forever (was: Some Blocks in Storage

[jira] [Created] (SPARK-19364) Some Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
Andrew Milkowski created SPARK-19364: Summary: Some Blocks in Storage Persists Forever Key: SPARK-19364 URL: https://issues.apache.org/jira/browse/SPARK-19364 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19364) Some Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Environment: ubuntu unix spark 2.0.2 application is java was: ubuntu unix spar 2.0.2

[jira] [Commented] (SPARK-11471) Improve the way that we plan shuffled join

2017-01-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836547#comment-15836547 ] Andrew Ash commented on SPARK-11471: [~yhuai] I'm interested in helping make progress on this -- it's

[jira] [Commented] (SPARK-16683) Group by does not work after multiple joins of the same dataframe

2017-01-20 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831878#comment-15831878 ] Andrew Ray commented on SPARK-16683: I'm working on a solution for this > Group by does not work

[jira] [Comment Edited] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-17 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827236#comment-15827236 ] Andrew MacKinlay edited comment on SPARK-19234 at 1/18/17 1:53 AM: ---

[jira] [Commented] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-17 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827236#comment-15827236 ] Andrew MacKinlay commented on SPARK-19234: -- [~yanbo] I presume that you are the author judging

[jira] [Updated] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-16 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew MacKinlay updated SPARK-19234: - Priority: Minor (was: Major) > AFTSurvivalRegression chokes silently or with confusing

[jira] [Commented] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-16 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824710#comment-15824710 ] Andrew MacKinlay commented on SPARK-19234: -- Ah, I think you're probably right about log of

[jira] [Updated] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-15 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew MacKinlay updated SPARK-19234: - Description: If you try and use AFTSurvivalRegression and any label in your input data

[jira] [Updated] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-15 Thread Andrew MacKinlay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew MacKinlay updated SPARK-19234: - Attachment: spark-aft-failure.txt Show failure case (a single zero-valued label, causing

[jira] [Created] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-15 Thread Andrew MacKinlay (JIRA)
Andrew MacKinlay created SPARK-19234: Summary: AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero Key: SPARK-19234 URL: https://issues.apache.org/jira/browse/SPARK-19234

[jira] [Updated] (SPARK-19213) FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the active session at execution time

2017-01-13 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19213: --- Summary: FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the

[jira] [Commented] (SPARK-18568) vertex attributes in the edge triplet not getting updated in super steps for Pregel API

2017-01-13 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822397#comment-15822397 ] Andrew Ray commented on SPARK-18568: RDD's have the same problem for cached collections of mutable

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-01-13 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821914#comment-15821914 ] Andrew Ray commented on SPARK-19116: The 2318 number is the size of the parquet files written to disk

[jira] [Commented] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

2017-01-13 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821868#comment-15821868 ] Andrew Ray commented on SPARK-19136: I forgot you can also just do: {code}

[jira] [Commented] (SPARK-8853) FPGrowth is not Java-Friendly

2017-01-10 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816049#comment-15816049 ] Andrew Ray commented on SPARK-8853: --- But there is no reason to directly create a {{FPGrowthModel}},

[jira] [Commented] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

2017-01-10 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815460#comment-15815460 ] Andrew Ray commented on SPARK-19136: You did not to a _typed_ aggregation so your result is a

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812819#comment-15812819 ] Andrew Ash commented on SPARK-18113: I've done some more diagnosis on an example I saw, and think

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811041#comment-15811041 ] Andrew Ash commented on SPARK-18113: Thanks for sending in that PR [~jinxing6...@126.com]! It's very

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802102#comment-15802102 ] Andrew Ash commented on SPARK-18113: [~xq2005] can you please send a PR to

[jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2016-12-28 Thread Andrew Snare (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783531#comment-15783531 ] Andrew Snare commented on SPARK-10878: -- I see this with Spark 2.0 as well. There doesn't appear to

[jira] [Commented] (SPARK-18393) DataFrame pivot output column names should respect aliases

2016-12-27 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781473#comment-15781473 ] Andrew Ray commented on SPARK-18393: It wouldn't hurt to backport to 2.0, its a pretty simple fix. >

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752907#comment-15752907 ] Andrew Ash commented on SPARK-18278: There are definitely challenges in building features that take

[jira] [Commented] (SPARK-17119) Add configuration property to allow the history server to delete .inprogress files

2016-12-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752439#comment-15752439 ] Andrew Ash commented on SPARK-17119: +1 I would use this feature > Add configuration property to

[jira] [Commented] (SPARK-18847) PageRank gives incorrect results for graphs with sinks

2016-12-13 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746395#comment-15746395 ] Andrew Ray commented on SPARK-18847: I have and have not found any relevant. I'm currently working on

[jira] [Commented] (SPARK-18845) PageRank has incorrect initialization value that leads to slow convergence

2016-12-13 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746385#comment-15746385 ] Andrew Ray commented on SPARK-18845: [~srowen] No that's a different thing just whether the result

[jira] [Created] (SPARK-18848) PageRank gives incorrect results for graphs with sinks

2016-12-13 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-18848: -- Summary: PageRank gives incorrect results for graphs with sinks Key: SPARK-18848 URL: https://issues.apache.org/jira/browse/SPARK-18848 Project: Spark Issue

[jira] [Created] (SPARK-18847) PageRank gives incorrect results for graphs with sinks

2016-12-13 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-18847: -- Summary: PageRank gives incorrect results for graphs with sinks Key: SPARK-18847 URL: https://issues.apache.org/jira/browse/SPARK-18847 Project: Spark Issue

[jira] [Created] (SPARK-18845) PageRank has incorrect initialization value that leads to slow convergence

2016-12-13 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-18845: -- Summary: PageRank has incorrect initialization value that leads to slow convergence Key: SPARK-18845 URL: https://issues.apache.org/jira/browse/SPARK-18845 Project:

[jira] [Updated] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-18113: --- Description: Executor sends *AskPermissionToCommitOutput* to driver failed, and retry another

[jira] [Updated] (SPARK-17664) Failed to saveAsHadoop when speculate is enabled

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-17664: --- Description: >From follow logs, task 22 has failed 4 times because of "the driver did not

[jira] [Commented] (SPARK-17859) persist should not impede with spark's ability to perform a broadcast join.

2016-12-08 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733247#comment-15733247 ] Andrew Ray commented on SPARK-17859: this appears to be fixed in 2.0.2 {code} scala>

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-08 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731562#comment-15731562 ] Andrew Ash commented on SPARK-18278: [~rxin] is it a problem for ASF projects to publish docker

[jira] [Commented] (SPARK-18717) Datasets - crash (compile exception) when mapping to immutable scala map

2016-12-05 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723527#comment-15723527 ] Andrew Ray commented on SPARK-18717: I have a fix for this, will make a PR in a bit > Datasets -

[jira] [Commented] (SPARK-18717) Datasets - crash (compile exception) when mapping to immutable scala map

2016-12-05 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723499#comment-15723499 ] Andrew Ray commented on SPARK-18717: Use `scala.collection.Map` as the type in your case class

[jira] [Commented] (SPARK-11705) Eliminate unnecessary Cartesian Join

2016-12-02 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715675#comment-15715675 ] Andrew Ray commented on SPARK-11705: Above example does not have a cartesian product in Spark 2.0.2

[jira] [Commented] (SPARK-18640) Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors

2016-12-02 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715503#comment-15715503 ] Andrew Or commented on SPARK-18640: --- my mistake > Fix minor synchronization issue in

[jira] [Commented] (SPARK-17896) Dataset groupByKey + reduceGroups fails with codegen-related exception

2016-11-28 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703320#comment-15703320 ] Andrew Ray commented on SPARK-17896: The given code seems to work in 2.0.2 > Dataset groupByKey +

[jira] [Reopened] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-11-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-17680: --- > Unicode Character Support for Column Names and Comments >

[jira] [Updated] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-11-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-17680: -- Assignee: Kazuaki Ishizaki > Unicode Character Support for Column Names and Comments >

[jira] [Resolved] (SPARK-17680) Unicode Character Support for Column Names and Comments

2016-11-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17680. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 > Unicode Character

[jira] [Resolved] (SPARK-18050) spark 2.0.1 enable hive throw AlreadyExistsException(message:Database default already exists)

2016-11-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-18050. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 > spark 2.0.1 enable

[jira] [Reopened] (SPARK-18050) spark 2.0.1 enable hive throw AlreadyExistsException(message:Database default already exists)

2016-11-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reopened SPARK-18050: --- Assignee: Wenchen Fan > spark 2.0.1 enable hive throw AlreadyExistsException(message:Database

[jira] [Resolved] (SPARK-18507) Major performance regression in SHOW PARTITIONS on partitioned Hive tables

2016-11-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-18507. --- Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.1.0 Target

[jira] [Resolved] (SPARK-18517) DROP TABLE IF EXISTS should not warn for non-existing tables

2016-11-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-18517. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.1.0 Target

[jira] [Resolved] (SPARK-18361) Expose RDD localCheckpoint in PySpark

2016-11-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-18361. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 > Expose RDD

[jira] [Updated] (SPARK-18361) Expose RDD localCheckpoint in PySpark

2016-11-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-18361: -- Assignee: Gabriel Huang > Expose RDD localCheckpoint in PySpark >

[jira] [Commented] (SPARK-18499) Add back support for custom Spark SQL dialects

2016-11-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675714#comment-15675714 ] Andrew Ash commented on SPARK-18499: Specifically what I'm most interested in is a strict ANSI SQL

[jira] [Created] (SPARK-18499) Add back support for custom Spark SQL dialects

2016-11-17 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-18499: -- Summary: Add back support for custom Spark SQL dialects Key: SPARK-18499 URL: https://issues.apache.org/jira/browse/SPARK-18499 Project: Spark Issue Type:

[jira] [Created] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

2016-11-15 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-18457: -- Summary: ORC and other columnar formats using HiveShim read all columns when doing a simple count Key: SPARK-18457 URL: https://issues.apache.org/jira/browse/SPARK-18457

[jira] [Created] (SPARK-18448) SparkSession should implement java.lang.AutoCloseable like JavaSparkContext

2016-11-15 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-18448: -- Summary: SparkSession should implement java.lang.AutoCloseable like JavaSparkContext Key: SPARK-18448 URL: https://issues.apache.org/jira/browse/SPARK-18448 Project:

[jira] [Resolved] (SPARK-17686) Propose to print Scala version in "spark-submit --version" command

2016-10-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17686. --- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.1.0 Target

[jira] [Resolved] (SPARK-17899) add a debug mode to keep raw table properties in HiveExternalCatalog

2016-10-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17899. --- Resolution: Fixed Fix Version/s: 2.1.0 > add a debug mode to keep raw table properties in

[jira] [Resolved] (SPARK-11272) Support importing and exporting event logs from HistoryServer web portal

2016-10-13 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-11272. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 Target

[jira] [Created] (SPARK-17874) Enabling SSL on HistoryServer should only open one port not two

2016-10-11 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-17874: -- Summary: Enabling SSL on HistoryServer should only open one port not two Key: SPARK-17874 URL: https://issues.apache.org/jira/browse/SPARK-17874 Project: Spark

[jira] [Updated] (SPARK-17874) Additional SSL port on HistoryServer should be configurable

2016-10-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-17874: --- Summary: Additional SSL port on HistoryServer should be configurable (was: Enabling SSL on

[jira] [Resolved] (SPARK-17715) Log INFO per task launch creates a large driver log

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17715. --- Resolution: Fixed Assignee: Brian Cho Fix Version/s: 2.1.0 Target

[jira] [Updated] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-17672: -- Assignee: Gang Wu > Spark 2.0 history server web Ui takes too long for a single application >

[jira] [Resolved] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17672. --- Resolution: Fixed Fix Version/s: 2.0.1 Target Version/s: 2.0.1 > Spark 2.0 history

[jira] [Resolved] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17648. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 >

[jira] [Resolved] (SPARK-17623) Failed tasks end reason is always a TaskFailedReason, types should reflect this

2016-09-21 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17623. --- Resolution: Fixed Fix Version/s: 2.1.0 > Failed tasks end reason is always a

[jira] [Resolved] (SPARK-17438) Master UI should show the correct core limit when `ApplicationInfo.executorLimit` is set

2016-09-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17438. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Master UI should show the

<    1   2   3   4   5   6   7   8   9   10   >