[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

2018-07-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541265#comment-16541265 ] Andrew Ash commented on SPARK-21962: Note that HTrace is now being removed from Hadoop – 

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347480#comment-16347480 ] Andrew Ash commented on SPARK-23274: Many thanks for the fast fix [~smilegator]! >

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-30 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345539#comment-16345539 ] Andrew Ash commented on SPARK-23274: Suspect this regression was introduced by

[jira] [Commented] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330943#comment-16330943 ] Andrew Ash commented on SPARK-22982: [~joshrosen] do you have some example stacktraces of what this

[jira] [Commented] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281354#comment-16281354 ] Andrew Ash commented on SPARK-22725: Demonstration of difference between {{.map}} on List vs Stream:

[jira] [Created] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22725: -- Summary: df.select on a Stream is broken, vs a List Key: SPARK-22725 URL: https://issues.apache.org/jira/browse/SPARK-22725 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

2017-11-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246328#comment-16246328 ] Andrew Ash commented on SPARK-22479: Completely agree that credentials shouldn't be in the toString

[jira] [Created] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22470: -- Summary: Doc that functions.hash is also used internally for shuffle and bucketing Key: SPARK-22470 URL: https://issues.apache.org/jira/browse/SPARK-22470 Project: Spark

[jira] [Commented] (SPARK-22042) ReorderJoinPredicates can break when child's partitioning is not decided

2017-10-26 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220734#comment-16220734 ] Andrew Ash commented on SPARK-22042: Hi I'm seeing this problem as well, thanks for investigating and

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

2017-10-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219139#comment-16219139 ] Andrew Ash commented on SPARK-21991: Thanks for the contribution to Spark [~nivox]! I'll be testing

[jira] [Commented] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

2017-10-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216529#comment-16216529 ] Andrew Ash commented on SPARK-21991: Thanks for debugging and diagnosing this [~nivox]! I'm seeing

[jira] [Commented] (SPARK-22204) Explain output for SQL with commands shows no optimization

2017-10-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208676#comment-16208676 ] Andrew Ash commented on SPARK-22204: One way to work around this issue could be by getting the child

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202795#comment-16202795 ] Andrew Ash commented on SPARK-22269: [~sowen] you closed this as a duplicate. What issue is it a

[jira] [Commented] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202793#comment-16202793 ] Andrew Ash commented on SPARK-22268: Any time {{./dev/run-tests}} is failing I consider that a bug.

[jira] [Created] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22269: -- Summary: Java style checks should be run in Jenkins Key: SPARK-22269 URL: https://issues.apache.org/jira/browse/SPARK-22269 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22268: -- Summary: Fix java style errors Key: SPARK-22268 URL: https://issues.apache.org/jira/browse/SPARK-22268 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-18359) Let user specify locale in CSV parsing

2017-10-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200831#comment-16200831 ] Andrew Ash commented on SPARK-18359: I agree with Sean -- using the submitting JVM's locale is

[jira] [Commented] (SPARK-20055) Documentation for CSV datasets in SQL programming guide

2017-10-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193355#comment-16193355 ] Andrew Ash commented on SPARK-20055: What I would find most useful is a list of available options and

[jira] [Created] (SPARK-22204) Explain output for SQL with commands shows no optimization

2017-10-04 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22204: -- Summary: Explain output for SQL with commands shows no optimization Key: SPARK-22204 URL: https://issues.apache.org/jira/browse/SPARK-22204 Project: Spark Issue

[jira] [Reopened] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2017-09-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-18016: // reopening issue One PR addressing this bug has been merged --

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-09-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178459#comment-16178459 ] Andrew Ash commented on SPARK-19700: There was a thread on the dev list recently about Apache Aurora:

[jira] [Created] (SPARK-22112) Add missing method to pyspark api: spark.read.csv(Dataset)

2017-09-24 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22112: -- Summary: Add missing method to pyspark api: spark.read.csv(Dataset) Key: SPARK-22112 URL: https://issues.apache.org/jira/browse/SPARK-22112 Project: Spark

[jira] [Created] (SPARK-21962) Distributed Tracing in Spark

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21962: -- Summary: Distributed Tracing in Spark Key: SPARK-21962 URL: https://issues.apache.org/jira/browse/SPARK-21962 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21953) Show both memory and disk bytes spilled if either is present

2017-09-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21953: -- Summary: Show both memory and disk bytes spilled if either is present Key: SPARK-21953 URL: https://issues.apache.org/jira/browse/SPARK-21953 Project: Spark

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2017-09-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156622#comment-16156622 ] Andrew Ash commented on SPARK-12449: [~velvia] I'm not involved with the CatalystSource or SAP

[jira] [Created] (SPARK-21941) Stop storing unused attemptId in SQLTaskMetrics

2017-09-07 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21941: -- Summary: Stop storing unused attemptId in SQLTaskMetrics Key: SPARK-21941 URL: https://issues.apache.org/jira/browse/SPARK-21941 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21807) The getAliasedConstraints function in LogicalPlan will take a long time when number of expressions is greater than 100

2017-08-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149600#comment-16149600 ] Andrew Ash commented on SPARK-21807: For reference, here's a stacktrace I'm seeing on a cluster

[jira] [Commented] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

2017-08-30 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147976#comment-16147976 ] Andrew Ash commented on SPARK-21875: I'd be interested in more details on why it can't be run in the

[jira] [Created] (SPARK-21875) Jenkins passes Java code that violates ./dev/lint-java

2017-08-30 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21875: -- Summary: Jenkins passes Java code that violates ./dev/lint-java Key: SPARK-21875 URL: https://issues.apache.org/jira/browse/SPARK-21875 Project: Spark Issue

[jira] [Commented] (SPARK-15689) Data source API v2

2017-08-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138509#comment-16138509 ] Andrew Ash commented on SPARK-15689: Can the authors of this document add a section contrasting the

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2017-08-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138501#comment-16138501 ] Andrew Ash commented on SPARK-12449: Relevant slides:

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136371#comment-16136371 ] Andrew Ash commented on SPARK-19552: I didn't see anything other than the issue you just commented on

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-20 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134657#comment-16134657 ] Andrew Ash commented on SPARK-19552: Heads up the next time someone attempts this: Upgrading to

[jira] [Updated] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

2017-08-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21757: --- Description: At the following two places if the {{e.reason}} value contains a single quote

[jira] [Created] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

2017-08-16 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21757: -- Summary: Jobs page fails to load when executor removed event's reason contains single quote Key: SPARK-21757 URL: https://issues.apache.org/jira/browse/SPARK-21757

[jira] [Commented] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-08-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122651#comment-16122651 ] Andrew Ash commented on SPARK-21564: [~irashid] a possible fix could look roughly like this:

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-08-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122549#comment-16122549 ] Andrew Ash commented on SPARK-21563: Thanks for the thoughts [~irashid] -- I submitted a PR

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120643#comment-16120643 ] Andrew Ash commented on SPARK-19116: Ah yes, for files it seems like Spark currently uses size of the

[jira] [Closed] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-19116. -- Resolution: Not A Problem > LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file >

[jira] [Commented] (SPARK-19116) LogicalPlan.statistics.sizeInBytes wrong for trivial parquet file

2017-08-04 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114751#comment-16114751 ] Andrew Ash commented on SPARK-19116: [~shea.parkes] does this answer your question? >

[jira] [Commented] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108117#comment-16108117 ] Andrew Ash commented on SPARK-20433: Sorry about not updating the ticket description -- the 2.6.7.1

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Description: There was a security vulnerability recently reported to the upstream jackson-databind

[jira] [Updated] (SPARK-20433) Update jackson-databind to 2.6.7.1

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Summary: Update jackson-databind to 2.6.7.1 (was: Update jackson-databind to 2.6.7) > Update

[jira] [Reopened] (SPARK-20433) Update jackson-databind to 2.6.7

2017-07-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash reopened SPARK-20433: > Update jackson-databind to 2.6.7 > > > Key:

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105951#comment-16105951 ] Andrew Ash commented on SPARK-20433: As I wrote in that PR, it's 2.6.7.1 of jackson-databind that has

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105772#comment-16105772 ] Andrew Ash commented on SPARK-21563: And for reference, I added this additional logging to assist in

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105751#comment-16105751 ] Andrew Ash commented on SPARK-20433: Here's the patch I put in my fork of Spark:

[jira] [Updated] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21564: --- Description: cc [~robert3005] I was seeing an issue where Spark was throwing this exception:

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21563: --- Description: cc [~robert3005] I was seeing this exception during some running Spark jobs:

[jira] [Created] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21564: -- Summary: TaskDescription decoding failure should fail the task Key: SPARK-21564 URL: https://issues.apache.org/jira/browse/SPARK-21564 Project: Spark Issue

[jira] [Created] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21563: -- Summary: Race condition when serializing TaskDescriptions and adding jars Key: SPARK-21563 URL: https://issues.apache.org/jira/browse/SPARK-21563 Project: Spark

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2017-07-25 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099584#comment-16099584 ] Andrew Ash commented on SPARK-14887: [~fang fang chen] have you seen this in the latest version of

[jira] [Commented] (SPARK-21289) Text and CSV formats do not support custom end-of-line delimiters

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081136#comment-16081136 ] Andrew Ash commented on SPARK-21289: Looks like this will fix SPARK-17227 also > Text and CSV

[jira] [Closed] (SPARK-15226) CSV file data-line with newline at first line load error

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-15226. -- Resolution: Fixed Fix Version/s: 2.2.0 Fixed by Fixed by

[jira] [Comment Edited] (SPARK-15226) CSV file data-line with newline at first line load error

2017-07-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081154#comment-16081154 ] Andrew Ash edited comment on SPARK-15226 at 7/10/17 9:07 PM: - Fixed by

[jira] [Created] (SPARK-21220) Use outputPartitioning's bucketing if possible on write

2017-06-26 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21220: -- Summary: Use outputPartitioning's bucketing if possible on write Key: SPARK-21220 URL: https://issues.apache.org/jira/browse/SPARK-21220 Project: Spark Issue

[jira] [Comment Edited] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058930#comment-16058930 ] Andrew Ash edited comment on SPARK-19700 at 6/22/17 7:47 AM: - Public

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058930#comment-16058930 ] Andrew Ash commented on SPARK-19700: Public implementation that's been around a while: Two Sigma's

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058917#comment-16058917 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Facebook's in-house

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-06-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041803#comment-16041803 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Nomad by [~barnardb] at

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

2017-06-02 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035171#comment-16035171 ] Andrew Ash commented on SPARK-20952: For the localProperties on SparkContext it does 2 things I can

[jira] [Created] (SPARK-20815) NullPointerException in RPackageUtils#checkManifestForR

2017-05-19 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-20815: -- Summary: NullPointerException in RPackageUtils#checkManifestForR Key: SPARK-20815 URL: https://issues.apache.org/jira/browse/SPARK-20815 Project: Spark Issue

[jira] [Commented] (SPARK-20683) Make table uncache chaining optional

2017-05-19 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018008#comment-16018008 ] Andrew Ash commented on SPARK-20683: Thanks for that diff [~shea.parkes] -- we're planning on trying

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

2017-05-16 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012011#comment-16012011 ] Andrew Ash commented on SPARK-19700: Found another potential implementation: Eagle cluster manager

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979630#comment-15979630 ] Andrew Ash commented on SPARK-20433: It's unclear if Spark is affected, I wanted to open this ticket

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Major (was: Blocker) > Security issue with jackson-databind >

[jira] [Updated] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20433: --- Priority: Critical (was: Major) > Security issue with jackson-databind >

[jira] [Created] (SPARK-20433) Security issue with jackson-databind

2017-04-21 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-20433: -- Summary: Security issue with jackson-databind Key: SPARK-20433 URL: https://issues.apache.org/jira/browse/SPARK-20433 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973763#comment-15973763 ] Andrew Ash commented on SPARK-20364: Thanks for the investigation [~hyukjin.kwon]! The proposal you

[jira] [Comment Edited] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

2017-04-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966782#comment-15966782 ] Andrew Ash edited comment on SPARK-1809 at 4/12/17 11:00 PM: - I'm not using

[jira] [Closed] (SPARK-1809) Mesos backend doesn't respect HADOOP_CONF_DIR

2017-04-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash closed SPARK-1809. - Resolution: Unresolved Not using Mesos anymore, so closing > Mesos backend doesn't respect

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2017-04-07 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961170#comment-15961170 ] Andrew Ash commented on SPARK-20144: This is a regression from 1.6 to the 2.x line. [~marmbrus]

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

2017-03-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939072#comment-15939072 ] Andrew Ash commented on SPARK-19372: I've seen this as well on parquet files. > Code generation for

[jira] [Updated] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-03-18 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19528: --- Description: when dynamic allocation is enabled, the external shuffle service is used for maintain

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda environment

[jira] [Updated] (SPARK-20001) Support PythonRunner executing inside a Conda env

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-20001: --- Description: Similar to SPARK-13587, I'm trying to allow the user to configure a Conda environment

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2017-03-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929591#comment-15929591 ] Andrew Ash commented on SPARK-18278: As an update on this ticket: For those not already aware, work

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-03-02 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893222#comment-15893222 ] Andrew Ash commented on SPARK-18113: We discovered another bug related to committing that causes task

[jira] [Commented] (SPARK-7354) Flaky test: o.a.s.deploy.SparkSubmitSuite --jars

2017-02-23 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881593#comment-15881593 ] Andrew Ash commented on SPARK-7354: --- We saw a flake for this test in the k8s repo's Travis builds too:

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865778#comment-15865778 ] Andrew Ash commented on SPARK-18113: [~xukun] the scenario you describe should be accommodated by the

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865536#comment-15865536 ] Andrew Ash commented on SPARK-18113: Thanks for the updates you both. I've been working with a

[jira] [Commented] (SPARK-19493) Remove Java 7 support

2017-02-10 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861576#comment-15861576 ] Andrew Ash commented on SPARK-19493: +1 -- we're removing Java 7 compatibility from core internal

[jira] [Commented] (SPARK-11471) Improve the way that we plan shuffled join

2017-01-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836547#comment-15836547 ] Andrew Ash commented on SPARK-11471: [~yhuai] I'm interested in helping make progress on this -- it's

[jira] [Updated] (SPARK-19213) FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the active session at execution time

2017-01-13 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-19213: --- Summary: FileSourceScanExec uses SparkSession from HadoopFsRelation creation time instead of the

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812819#comment-15812819 ] Andrew Ash commented on SPARK-18113: I've done some more diagnosis on an example I saw, and think

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-09 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811041#comment-15811041 ] Andrew Ash commented on SPARK-18113: Thanks for sending in that PR [~jinxing6...@126.com]! It's very

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-01-05 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802102#comment-15802102 ] Andrew Ash commented on SPARK-18113: [~xq2005] can you please send a PR to

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752907#comment-15752907 ] Andrew Ash commented on SPARK-18278: There are definitely challenges in building features that take

[jira] [Commented] (SPARK-17119) Add configuration property to allow the history server to delete .inprogress files

2016-12-15 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752439#comment-15752439 ] Andrew Ash commented on SPARK-17119: +1 I would use this feature > Add configuration property to

[jira] [Updated] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-18113: --- Description: Executor sends *AskPermissionToCommitOutput* to driver failed, and retry another

[jira] [Updated] (SPARK-17664) Failed to saveAsHadoop when speculate is enabled

2016-12-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-17664: --- Description: >From follow logs, task 22 has failed 4 times because of "the driver did not

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-08 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731562#comment-15731562 ] Andrew Ash commented on SPARK-18278: [~rxin] is it a problem for ASF projects to publish docker

[jira] [Commented] (SPARK-18499) Add back support for custom Spark SQL dialects

2016-11-17 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675714#comment-15675714 ] Andrew Ash commented on SPARK-18499: Specifically what I'm most interested in is a strict ANSI SQL

[jira] [Created] (SPARK-18499) Add back support for custom Spark SQL dialects

2016-11-17 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-18499: -- Summary: Add back support for custom Spark SQL dialects Key: SPARK-18499 URL: https://issues.apache.org/jira/browse/SPARK-18499 Project: Spark Issue Type:

[jira] [Created] (SPARK-18448) SparkSession should implement java.lang.AutoCloseable like JavaSparkContext

2016-11-15 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-18448: -- Summary: SparkSession should implement java.lang.AutoCloseable like JavaSparkContext Key: SPARK-18448 URL: https://issues.apache.org/jira/browse/SPARK-18448 Project:

[jira] [Created] (SPARK-17874) Enabling SSL on HistoryServer should only open one port not two

2016-10-11 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-17874: -- Summary: Enabling SSL on HistoryServer should only open one port not two Key: SPARK-17874 URL: https://issues.apache.org/jira/browse/SPARK-17874 Project: Spark

[jira] [Updated] (SPARK-17874) Additional SSL port on HistoryServer should be configurable

2016-10-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-17874: --- Summary: Additional SSL port on HistoryServer should be configurable (was: Enabling SSL on

[jira] [Commented] (SPARK-17227) Allow configuring record delimiter in csv

2016-08-24 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435539#comment-15435539 ] Andrew Ash commented on SPARK-17227: Rob and I work together, and we've seen datasets in mostly-CSV

[jira] [Commented] (SPARK-17029) Dataset toJSON goes through RDD form instead of transforming dataset itself

2016-08-11 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418149#comment-15418149 ] Andrew Ash commented on SPARK-17029: Note RDD form usage from

[jira] [Created] (SPARK-15104) Bad spacing in log line

2016-05-03 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-15104: -- Summary: Bad spacing in log line Key: SPARK-15104 URL: https://issues.apache.org/jira/browse/SPARK-15104 Project: Spark Issue Type: Bug Affects Versions:

  1   2   3   4   >