[jira] [Updated] (SPARK-19160) Decorator for UDF creation.

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19160: --- Affects Version/s: (was: 1.5.0) > Decorator for UDF creation. >

[jira] [Updated] (SPARK-19159) PySpark UDF API improvements

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19159: --- Affects Version/s: (was: 1.5.0) > PySpark UDF API improvements >

[jira] [Created] (SPARK-19164) Review of UserDefinedFunction._broadcast

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19164: -- Summary: Review of UserDefinedFunction._broadcast Key: SPARK-19164 URL: https://issues.apache.org/jira/browse/SPARK-19164 Project: Spark Issue

[jira] [Updated] (SPARK-19160) Decorator for UDF creation.

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19160: --- Affects Version/s: 2.2.0 > Decorator for UDF creation. > ---

[jira] [Resolved] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

2017-01-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-18997. Resolution: Fixed Assignee: Sean Owen Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19159) PySpark UDF API improvements

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19159: --- Affects Version/s: 2.2.0 > PySpark UDF API improvements >

[jira] [Created] (SPARK-19163) Lazy creation of the _judf

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19163: -- Summary: Lazy creation of the _judf Key: SPARK-19163 URL: https://issues.apache.org/jira/browse/SPARK-19163 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-19162) Input types validation

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19162: -- Summary: Input types validation Key: SPARK-19162 URL: https://issues.apache.org/jira/browse/SPARK-19162 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-19160) Decorator for UDF creation.

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-19160: --- Summary: Decorator for UDF creation. (was: UDF creation) > Decorator for UDF

[jira] [Created] (SPARK-19161) Improving UDF Docstrings

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19161: -- Summary: Improving UDF Docstrings Key: SPARK-19161 URL: https://issues.apache.org/jira/browse/SPARK-19161 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-19160) UDF creation

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19160: -- Summary: UDF creation Key: SPARK-19160 URL: https://issues.apache.org/jira/browse/SPARK-19160 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-18871) New test cases for IN/NOT IN subquery

2017-01-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816068#comment-15816068 ] Reynold Xin commented on SPARK-18871: - Yes I mean just submit them under this jira. > New test

[jira] [Created] (SPARK-19159) PySpark UDF API improvements

2017-01-10 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-19159: -- Summary: PySpark UDF API improvements Key: SPARK-19159 URL: https://issues.apache.org/jira/browse/SPARK-19159 Project: Spark Issue Type:

[jira] [Commented] (SPARK-8853) FPGrowth is not Java-Friendly

2017-01-10 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816049#comment-15816049 ] Andrew Ray commented on SPARK-8853: --- But there is no reason to directly create a {{FPGrowthModel}},

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816026#comment-15816026 ] Mridul Muralidharan commented on SPARK-19143: - bq. You say "we added", you are saying you

[jira] [Created] (SPARK-19158) ml.R example fails in yarn-cluster mode due to lacks of e1071 package

2017-01-10 Thread Yesha Vora (JIRA)
Yesha Vora created SPARK-19158: -- Summary: ml.R example fails in yarn-cluster mode due to lacks of e1071 package Key: SPARK-19158 URL: https://issues.apache.org/jira/browse/SPARK-19158 Project: Spark

[jira] [Comment Edited] (SPARK-19145) Timestamp to String casting is slowing the query significantly

2017-01-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815992#comment-15815992 ] Dongjoon Hyun edited comment on SPARK-19145 at 1/10/17 7:52 PM: Hi,

[jira] [Updated] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

2017-01-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19113: - Fix Version/s: 2.1.1 > Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a

[jira] [Commented] (SPARK-19145) Timestamp to String casting is slowing the query significantly

2017-01-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815992#comment-15815992 ] Dongjoon Hyun commented on SPARK-19145: --- Hi, [~tanejagagan]. You can start to work by yourself.

[jira] [Commented] (SPARK-19133) SparkR glm Gamma family results in error

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815986#comment-15815986 ] Apache Spark commented on SPARK-19133: -- User 'felixcheung' has created a pull request for this

[jira] [Resolved] (SPARK-19133) SparkR glm Gamma family results in error

2017-01-10 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-19133. -- Resolution: Fixed Fix Version/s: 2.2.0 Target Version/s: 2.2.0 > SparkR glm

[jira] [Commented] (SPARK-17993) Spark prints an avalanche of warning messages from Parquet when reading parquet files written by older versions of Parquet-mr

2017-01-10 Thread Emre Colak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815967#comment-15815967 ] Emre Colak commented on SPARK-17993: Hi Michael, Thanks for getting back. I see "parquet-mr version

[jira] [Commented] (SPARK-17993) Spark prints an avalanche of warning messages from Parquet when reading parquet files written by older versions of Parquet-mr

2017-01-10 Thread Michael Allman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815965#comment-15815965 ] Michael Allman commented on SPARK-17993: Hi Emre, Thanks for reporting this. To clarify, what do

[jira] [Updated] (SPARK-18917) Dataframe - Time Out Issues / Taking long time in append mode on object stores

2017-01-10 Thread Anbu Cheeralan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated SPARK-18917: --- Affects Version/s: 2.1.0 > Dataframe - Time Out Issues / Taking long time in append mode on

[jira] [Comment Edited] (SPARK-18917) Dataframe - Time Out Issues / Taking long time in append mode on object stores

2017-01-10 Thread Anbu Cheeralan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812608#comment-15812608 ] Anbu Cheeralan edited comment on SPARK-18917 at 1/10/17 7:31 PM: - I agree

[jira] [Commented] (SPARK-18871) New test cases for IN/NOT IN subquery

2017-01-10 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815876#comment-15815876 ] kevin yu commented on SPARK-18871: -- Hello Reyold: Sorry, I misunderstood your comment. The pr16337 is

[jira] [Resolved] (SPARK-19137) Garbage left in source tree after SQL tests are run

2017-01-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19137. -- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

2017-01-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815815#comment-15815815 ] Shixiong Zhu commented on SPARK-18905: -- Sure. Please go ahead. > Potential Issue of Semantics of

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815643#comment-15815643 ] Marcelo Vanzin commented on SPARK-19143: bq. My initial thought was to create an rpc between the

[jira] [Commented] (SPARK-17993) Spark prints an avalanche of warning messages from Parquet when reading parquet files written by older versions of Parquet-mr

2017-01-10 Thread Emre Colak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815632#comment-15815632 ] Emre Colak commented on SPARK-17993: I'm using spark-shell in version 2.1 and still see this issue

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-10 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815580#comment-15815580 ] nirav patel commented on SPARK-19090: - I tried passing spark parameters via oozie directly using

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-10 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815510#comment-15815510 ] nirav patel commented on SPARK-19090: - Yes, I am setting it from a Main application class. It works

[jira] [Assigned] (SPARK-19157) should be able to change spark.sql.runSQLOnFiles at runtime

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19157: Assignee: Apache Spark (was: Wenchen Fan) > should be able to change

[jira] [Commented] (SPARK-19157) should be able to change spark.sql.runSQLOnFiles at runtime

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815492#comment-15815492 ] Apache Spark commented on SPARK-19157: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19157) should be able to change spark.sql.runSQLOnFiles at runtime

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19157: Assignee: Wenchen Fan (was: Apache Spark) > should be able to change

[jira] [Created] (SPARK-19157) should be able to change spark.sql.runSQLOnFiles at runtime

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19157: --- Summary: should be able to change spark.sql.runSQLOnFiles at runtime Key: SPARK-19157 URL: https://issues.apache.org/jira/browse/SPARK-19157 Project: Spark

[jira] [Assigned] (SPARK-18929) Add Tweedie distribution in GLM

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18929: Assignee: Wayne Zhang (was: Apache Spark) > Add Tweedie distribution in GLM >

[jira] [Assigned] (SPARK-18929) Add Tweedie distribution in GLM

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18929: Assignee: Apache Spark (was: Wayne Zhang) > Add Tweedie distribution in GLM >

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815468#comment-15815468 ] Sean Owen commented on SPARK-19156: --- It's probably because you wrote this in a non-static method of an

[jira] [Commented] (SPARK-16367) Wheelhouse Support for PySpark

2017-01-10 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815466#comment-15815466 ] Semet commented on SPARK-16367: --- Yes in our pull request both conda and pip are supported. Wheel allow pip

[jira] [Commented] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

2017-01-10 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815460#comment-15815460 ] Andrew Ray commented on SPARK-19136: You did not to a _typed_ aggregation so your result is a

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-10 Thread Rafael Guglielmetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815440#comment-15815440 ] Rafael Guglielmetti commented on SPARK-19156: - [~srowen] Thanks! Unfortunately I'm a

[jira] [Created] (SPARK-19156) Example in the doc not working

2017-01-10 Thread Rafael Guglielmetti (JIRA)
Rafael Guglielmetti created SPARK-19156: --- Summary: Example in the doc not working Key: SPARK-19156 URL: https://issues.apache.org/jira/browse/SPARK-19156 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815412#comment-15815412 ] Sean Owen commented on SPARK-19156: --- Oops, you're right. Feel free to open a PR; the content is at

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2017-01-10 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815352#comment-15815352 ] Ruslan Dautkhanov commented on SPARK-5493: -- Thanks again [~vanzin] for the feedback, please check

[jira] [Reopened] (SPARK-18929) Add Tweedie distribution in GLM

2017-01-10 Thread Wayne Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wayne Zhang reopened SPARK-18929: - > Add Tweedie distribution in GLM > --- > > Key:

[jira] [Commented] (SPARK-12076) countDistinct behaves inconsistently

2017-01-10 Thread Paul Zaczkieiwcz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815268#comment-15815268 ] Paul Zaczkieiwcz commented on SPARK-12076: -- Actually, the counts were wrong with one of the

[jira] [Resolved] (SPARK-2827) Add DegreeDist function support

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2827. -- Resolution: Won't Fix > Add DegreeDist function support > --- > >

[jira] [Resolved] (SPARK-13607) Improves compression performance for integer-typed values on cache to reduce GC pressure

2017-01-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-13607. -- Resolution: Won't Fix I think this improvement is not always necessary, so I'll close

[jira] [Issue Comment Deleted] (SPARK-19133) SparkR glm Gamma family results in error

2017-01-10 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-19133: Comment: was deleted (was: User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-19155) ML estimator string params should support both uppercase and lowercase

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815175#comment-15815175 ] Apache Spark commented on SPARK-19155: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19155) ML estimator string params should support both uppercase and lowercase

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19155: Assignee: (was: Apache Spark) > ML estimator string params should support both

[jira] [Assigned] (SPARK-19155) ML estimator string params should support both uppercase and lowercase

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19155: Assignee: Apache Spark > ML estimator string params should support both uppercase and

[jira] [Created] (SPARK-19155) ML estimator string params should support both uppercase and lowercase

2017-01-10 Thread Yanbo Liang (JIRA)
Yanbo Liang created SPARK-19155: --- Summary: ML estimator string params should support both uppercase and lowercase Key: SPARK-19155 URL: https://issues.apache.org/jira/browse/SPARK-19155 Project: Spark

[jira] [Commented] (SPARK-14165) NoSuchElementException: None.get when joining DataFrames with Seq of fields of different case

2017-01-10 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815105#comment-15815105 ] Jacek Laskowski commented on SPARK-14165: - It can be closed but with FIXED resolution since it

[jira] [Resolved] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19113. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16492

[jira] [Commented] (SPARK-16367) Wheelhouse Support for PySpark

2017-01-10 Thread Benjamin Zaitlen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815083#comment-15815083 ] Benjamin Zaitlen commented on SPARK-16367: -- [~gae...@xeberon.net] do you also have plans on

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815078#comment-15815078 ] Thomas Graves commented on SPARK-19143: --- [~mridulm80] You say "we added", you are saying you have

[jira] [Assigned] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18997: Assignee: Apache Spark > Recommended upgrade libthrift to 0.9.3 >

[jira] [Assigned] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18997: Assignee: (was: Apache Spark) > Recommended upgrade libthrift to 0.9.3 >

[jira] [Commented] (SPARK-18997) Recommended upgrade libthrift to 0.9.3

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815007#comment-15815007 ] Apache Spark commented on SPARK-18997: -- User 'srowen' has created a pull request for this issue:

[jira] [Commented] (SPARK-15034) Use the value of spark.sql.warehouse.dir as the warehouse location instead of using hive.metastore.warehouse.dir

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814975#comment-15814975 ] Sean Owen commented on SPARK-15034: --- It's mentioned in the sql-programming-guide.html doc > Use the

[jira] [Resolved] (SPARK-18857) SparkSQL ThriftServer hangs while extracting huge data volumes in incremental collect mode

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18857. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16440

[jira] [Updated] (SPARK-18857) SparkSQL ThriftServer hangs while extracting huge data volumes in incremental collect mode

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18857: -- Assignee: Dongjoon Hyun > SparkSQL ThriftServer hangs while extracting huge data volumes in

[jira] [Resolved] (SPARK-19117) script transformation does not work on Windows due to fixed bash executable location

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19117. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16501

[jira] [Updated] (SPARK-19117) script transformation does not work on Windows due to fixed bash executable location

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19117: -- Assignee: Hyukjin Kwon > script transformation does not work on Windows due to fixed bash executable

[jira] [Resolved] (SPARK-18922) Fix more resource-closing-related and path-related test failures in identified ones on Windows

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18922. --- Resolution: Fixed Fix Version/s: 2.2.0 > Fix more resource-closing-related and path-related

[jira] [Updated] (SPARK-19154) support read and overwrite a same table

2017-01-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19154: Description: In SPARK-5746 , we forbid users to read and overwrite a same table. It seems like we

[jira] [Resolved] (SPARK-16766) TakeOrderedAndProjectExec easily cause OOM

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16766. --- Resolution: Won't Fix > TakeOrderedAndProjectExec easily cause OOM >

[jira] [Resolved] (SPARK-17081) Empty strings not preserved which causes SQLException: mismatching column value count

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17081. --- Resolution: Cannot Reproduce Fix Version/s: 2.0.0 > Empty strings not preserved which causes

[jira] [Assigned] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19149: Assignee: (was: Apache Spark) > Unify two sets of statistics in LogicalPlan >

[jira] [Commented] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814897#comment-15814897 ] Apache Spark commented on SPARK-19149: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Updated] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-19149: - Description: Currently we have two sets of statistics in LogicalPlan: a simple stats and a stats

[jira] [Created] (SPARK-19154) support read and overwrite a same table

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19154: --- Summary: support read and overwrite a same table Key: SPARK-19154 URL: https://issues.apache.org/jira/browse/SPARK-19154 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19149: Assignee: Apache Spark > Unify two sets of statistics in LogicalPlan >

[jira] [Updated] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-19149: - Description: We have two sets of statistics in LogicalPlan: a simple stats and a stats estimated

[jira] [Updated] (SPARK-19151) DataFrameWriter.saveAsTable should work with hive format with overwrite mode

2017-01-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19151: Target Version/s: 2.2.0 > DataFrameWriter.saveAsTable should work with hive format with overwrite

[jira] [Created] (SPARK-19153) DataFrameWriter.saveAsTable should work with hive format to create partitioned table

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19153: --- Summary: DataFrameWriter.saveAsTable should work with hive format to create partitioned table Key: SPARK-19153 URL: https://issues.apache.org/jira/browse/SPARK-19153

[jira] [Created] (SPARK-19152) DataFrameWriter.saveAsTable should work with hive format with append mode

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19152: --- Summary: DataFrameWriter.saveAsTable should work with hive format with append mode Key: SPARK-19152 URL: https://issues.apache.org/jira/browse/SPARK-19152 Project:

[jira] [Created] (SPARK-19151) DataFrameWriter.saveAsTable should work with hive format with overwrite mode

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19151: --- Summary: DataFrameWriter.saveAsTable should work with hive format with overwrite mode Key: SPARK-19151 URL: https://issues.apache.org/jira/browse/SPARK-19151 Project:

[jira] [Created] (SPARK-19149) Unify two sets of statistics in LogicalPlan

2017-01-10 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-19149: Summary: Unify two sets of statistics in LogicalPlan Key: SPARK-19149 URL: https://issues.apache.org/jira/browse/SPARK-19149 Project: Spark Issue Type:

[jira] [Created] (SPARK-19150) completely support using hive as data source to create tables

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19150: --- Summary: completely support using hive as data source to create tables Key: SPARK-19150 URL: https://issues.apache.org/jira/browse/SPARK-19150 Project: Spark

[jira] [Commented] (SPARK-19148) do not expose the external table concept in Catalog

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814865#comment-15814865 ] Apache Spark commented on SPARK-19148: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19148) do not expose the external table concept in Catalog

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19148: Assignee: Wenchen Fan (was: Apache Spark) > do not expose the external table concept in

[jira] [Assigned] (SPARK-19148) do not expose the external table concept in Catalog

2017-01-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19148: Assignee: Apache Spark (was: Wenchen Fan) > do not expose the external table concept in

[jira] [Created] (SPARK-19148) do not expose the external table concept in Catalog

2017-01-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19148: --- Summary: do not expose the external table concept in Catalog Key: SPARK-19148 URL: https://issues.apache.org/jira/browse/SPARK-19148 Project: Spark Issue

[jira] [Resolved] (SPARK-14660) Executors show up active tasks indefinitely after stage is killed

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14660. --- Resolution: Duplicate > Executors show up active tasks indefinitely after stage is killed >

[jira] [Resolved] (SPARK-14309) Dataframe returns wrong results due to parsing incorrectly

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14309. --- Resolution: Duplicate > Dataframe returns wrong results due to parsing incorrectly >

[jira] [Commented] (SPARK-14165) NoSuchElementException: None.get when joining DataFrames with Seq of fields of different case

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814823#comment-15814823 ] Sean Owen commented on SPARK-14165: --- Either the description needs to be edited to reflect the actual

[jira] [Resolved] (SPARK-12809) Spark SQL UDF does not work with struct input parameters

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12809. --- Resolution: Duplicate > Spark SQL UDF does not work with struct input parameters >

[jira] [Resolved] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2017-01-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18112. --- Resolution: Duplicate > Spark2.x does not support read data from Hive 2.x metastore >

[jira] [Updated] (SPARK-19147) netty throw NPE

2017-01-10 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19147: -- Summary: netty throw NPE (was: netty throw NPE, can not found group) > netty throw NPE >

[jira] [Resolved] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2017-01-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16845. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue resolved by pull

[jira] [Updated] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2017-01-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-16845: Assignee: Liwei Lin > org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering"

[jira] [Updated] (SPARK-19147) netty throw NPE, can not found group

2017-01-10 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19147: -- Description: {code} 17/01/10 19:17:20 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from

[jira] [Commented] (SPARK-19147) netty throw NPE, can not found group

2017-01-10 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814720#comment-15814720 ] cen yuhai commented on SPARK-19147: --- [~zsxwing] > netty throw NPE, can not found group >

[jira] [Created] (SPARK-19147) netty throw NPE, can not found group

2017-01-10 Thread cen yuhai (JIRA)
cen yuhai created SPARK-19147: - Summary: netty throw NPE, can not found group Key: SPARK-19147 URL: https://issues.apache.org/jira/browse/SPARK-19147 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-19107) support creating hive table with DataFrameWriter and Catalog

2017-01-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19107. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16487

[jira] [Updated] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Description: The performance of the

[jira] [Updated] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Description: The performance of the

<    1   2   3   >