[jira] [Updated] (SPARK-18590) R - Include package vignettes and help pages, build source package in Spark distribution

[ https://issues.apache.org/jira/browse/SPARK-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18590: Target Version/s: 2.1.0 > R - Include package vignettes and help pages, build source package in Spa

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17949: Labels: releasenotes (was: ) > Introduce a JVM object based aggregate operator > -

[jira] [Updated] (SPARK-14543) SQL/Hive insertInto has unexpected results

[ https://issues.apache.org/jira/browse/SPARK-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-14543: Target Version/s: 2.2.0 (was: 2.1.0) > SQL/Hive insertInto has unexpected results > --

[jira] [Commented] (SPARK-17897) not isnotnull is converted to the always false condition isnotnull && not isnotnull

[ https://issues.apache.org/jira/browse/SPARK-17897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704335#comment-15704335 ] Reynold Xin commented on SPARK-17897: - cc [~cloud_fan], [~smilegator], [~hvanhovell]

[jira] [Updated] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

[ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18544: Issue Type: Sub-task (was: Bug) Parent: SPARK-17861 > Append with df.saveAsTable writes da

[jira] [Updated] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

[ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18544: Assignee: Eric Liang > Append with df.saveAsTable writes data to wrong location > -

[jira] [Resolved] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

[ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18544. - Resolution: Fixed Fix Version/s: 2.1.0 > Append with df.saveAsTable writes data to wrong l

[jira] [Resolved] (SPARK-18523) OOM killer may leave SparkContext in broken state causing Connection Refused errors

[ https://issues.apache.org/jira/browse/SPARK-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18523. - Resolution: Fixed Assignee: Alexander Shorin Fix Version/s: 2.1.0 > OOM killer ma

[jira] [Resolved] (SPARK-18585) Use `ev.isNull = "false"` if possible for Janino to have a chance to optimize.

[ https://issues.apache.org/jira/browse/SPARK-18585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18585. - Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 2.1.0 > Use `ev.isNull =

[jira] [Resolved] (SPARK-18482) make sure Spark can access the table metadata created by older version of spark

[ https://issues.apache.org/jira/browse/SPARK-18482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18482. - Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.1.0 > make sure Spark ca

[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods

[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700108#comment-15700108 ] Reynold Xin commented on SPARK-18120: - They should get triggered. Are you sure they d

[jira] [Resolved] (SPARK-18583) Fix nullability of InputFileName.

[ https://issues.apache.org/jira/browse/SPARK-18583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18583. - Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 2.1.0 > Fix nullability

[jira] [Commented] (SPARK-18487) Add task completion listener to HashAggregate to avoid memory leak

[ https://issues.apache.org/jira/browse/SPARK-18487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15696458#comment-15696458 ] Reynold Xin commented on SPARK-18487: - As discussed on the pull request, this is not

[jira] [Closed] (SPARK-18487) Add task completion listener to HashAggregate to avoid memory leak

[ https://issues.apache.org/jira/browse/SPARK-18487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18487. --- Resolution: Not A Problem > Add task completion listener to HashAggregate to avoid memory leak >

[jira] [Closed] (SPARK-17786) [SPARK 2.0] Sorting algorithm gives higher skewness of output

[ https://issues.apache.org/jira/browse/SPARK-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-17786. --- Resolution: Not A Problem Closing this. Please use the dev list for discussion. > [SPARK 2.0] Sorti

[jira] [Commented] (SPARK-18482) make sure Spark can access the table metadata created by older version of spark

[ https://issues.apache.org/jira/browse/SPARK-18482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691450#comment-15691450 ] Reynold Xin commented on SPARK-18482: - What's the deal with this? Can we just dump th

[jira] [Updated] (SPARK-17786) [SPARK 2.0] Sorting algorithm gives higher skewness of output

[ https://issues.apache.org/jira/browse/SPARK-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17786: Target Version/s: (was: 2.1.0) > [SPARK 2.0] Sorting algorithm gives higher skewness of output >

[jira] [Updated] (SPARK-17910) Allow users to update the comment of a column

[ https://issues.apache.org/jira/browse/SPARK-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17910: Target Version/s: 2.2.0 (was: 2.1.0) > Allow users to update the comment of a column > ---

[jira] [Created] (SPARK-18557) Downgrade the memory leak warning message

Reynold Xin created SPARK-18557: --- Summary: Downgrade the memory leak warning message Key: SPARK-18557 URL: https://issues.apache.org/jira/browse/SPARK-18557 Project: Spark Issue Type: Improveme

[jira] [Closed] (SPARK-8966) Design a mechanism to ensure that temporary files created in tasks are cleaned up after failures

[ https://issues.apache.org/jira/browse/SPARK-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-8966. -- Resolution: Later > Design a mechanism to ensure that temporary files created in tasks are > cleaned up

[jira] [Resolved] (SPARK-18179) Throws analysis exception with a proper message for unsupported argument types in reflect/java_method function

[ https://issues.apache.org/jira/browse/SPARK-18179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18179. - Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.1.0 > Throws analysis e

[jira] [Closed] (SPARK-13649) Move CalendarInterval out of unsafe package

[ https://issues.apache.org/jira/browse/SPARK-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-13649. --- Resolution: Won't Fix Target Version/s: (was: 2.1.0) > Move CalendarInterval out of unsaf

[jira] [Comment Edited] (SPARK-12469) Data Property Accumulators for Spark (formerly Consistent Accumulators)

[ https://issues.apache.org/jira/browse/SPARK-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687668#comment-15687668 ] Reynold Xin edited comment on SPARK-12469 at 11/22/16 7:36 PM:

[jira] [Commented] (SPARK-12469) Data Property Accumulators for Spark (formerly Consistent Accumulators)

[ https://issues.apache.org/jira/browse/SPARK-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687668#comment-15687668 ] Reynold Xin commented on SPARK-12469: - Sorry there is no way to get this in 2.1, give

[jira] [Resolved] (SPARK-17765) org.apache.spark.mllib.linalg.VectorUDT cannot be cast to org.apache.spark.sql.types.StructType

[ https://issues.apache.org/jira/browse/SPARK-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17765. - Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.1.0 > org.apache.spark.

[jira] [Created] (SPARK-18522) Create explicit contract for column stats serialization

Reynold Xin created SPARK-18522: --- Summary: Create explicit contract for column stats serialization Key: SPARK-18522 URL: https://issues.apache.org/jira/browse/SPARK-18522 Project: Spark Issue T

[jira] [Updated] (SPARK-15214) Implement code generation for Generate

[ https://issues.apache.org/jira/browse/SPARK-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15214: Summary: Implement code generation for Generate (was: Enable code generation for Generate) > Impl

[jira] [Resolved] (SPARK-15214) Enable code generation for Generate

[ https://issues.apache.org/jira/browse/SPARK-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15214. - Resolution: Fixed Fix Version/s: 2.2.0 Target Version/s: (was: 2.1.0) > Enable

[jira] [Resolved] (SPARK-18508) Fix documentation for DateDiff

[ https://issues.apache.org/jira/browse/SPARK-18508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18508. - Resolution: Fixed Fix Version/s: 2.1.0 > Fix documentation for DateDiff >

[jira] [Resolved] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18458. - Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.1.0 > core dumped r

[jira] [Commented] (SPARK-18510) Partition schema inference corrupts data

[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680487#comment-15680487 ] Reynold Xin commented on SPARK-18510: - Does your pull request for SPARK-18407 fix thi

[jira] [Created] (SPARK-18508) Fix documentation for DateDiff

Reynold Xin created SPARK-18508: --- Summary: Fix documentation for DateDiff Key: SPARK-18508 URL: https://issues.apache.org/jira/browse/SPARK-18508 Project: Spark Issue Type: Bug Compon

[jira] [Resolved] (SPARK-18505) Simplify AnalyzeColumnCommand

[ https://issues.apache.org/jira/browse/SPARK-18505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18505. - Resolution: Fixed Fix Version/s: 2.1.0 > Simplify AnalyzeColumnCommand > -

[jira] [Closed] (SPARK-18000) Aggregation function for computing bins (distinct value, count) pairs for equi-width histograms

[ https://issues.apache.org/jira/browse/SPARK-18000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18000. --- Resolution: Won't Fix Marking this as won't fix, since it looks like combination of count-min sketch

[jira] [Updated] (SPARK-18505) Simplify AnalyzeColumnCommand

[ https://issues.apache.org/jira/browse/SPARK-18505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18505: Description: I'm spending more time at the design & code level for cost-based optimizer now, and h

[jira] [Created] (SPARK-18505) Simplify AnalyzeColumnCommand

Reynold Xin created SPARK-18505: --- Summary: Simplify AnalyzeColumnCommand Key: SPARK-18505 URL: https://issues.apache.org/jira/browse/SPARK-18505 Project: Spark Issue Type: Sub-task Co

[jira] [Closed] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18252. --- Resolution: Won't Fix > Improve serialized BloomFilter size > --- > >

[jira] [Commented] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677521#comment-15677521 ] Reynold Xin commented on SPARK-18252: - Thanks - going to close this. > Improve seri

[jira] [Resolved] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

[ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18457. - Resolution: Fixed Assignee: Andrew Ray Fix Version/s: 2.1.0 > ORC and other colum

[jira] [Updated] (SPARK-18478) Support codegen for Hive UDFs

[ https://issues.apache.org/jira/browse/SPARK-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18478: Target Version/s: 2.2.0 > Support codegen for Hive UDFs > - > >

[jira] [Resolved] (SPARK-18462) SparkListenerDriverAccumUpdates event does not deserialize properly in history server

[ https://issues.apache.org/jira/browse/SPARK-18462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18462. - Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > SparkListenerDriverAccum

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675517#comment-15675517 ] Reynold Xin commented on SPARK-18352: - Actually just talked to [~marmbrus] and now I

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675446#comment-15675446 ] Reynold Xin commented on SPARK-18352: - No that's not sufficient. It doesn't do stream

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675437#comment-15675437 ] Reynold Xin commented on SPARK-18352: - I guess maybe it should be a user-configurable

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675405#comment-15675405 ] Reynold Xin commented on SPARK-18352: - Are these actually record delimiters? If the t

[jira] [Commented] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674801#comment-15674801 ] Reynold Xin commented on SPARK-18252: - Those two methods are pretty inefficient. Whe

[jira] [Commented] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674818#comment-15674818 ] Reynold Xin commented on SPARK-18252: - Regarding this - can you find some performance

[jira] [Commented] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674767#comment-15674767 ] Reynold Xin commented on SPARK-18252: - For 3, the sketch package has no external depe

[jira] [Commented] (SPARK-18252) Improve serialized BloomFilter size

[ https://issues.apache.org/jira/browse/SPARK-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674532#comment-15674532 ] Reynold Xin commented on SPARK-18252: - I'm not sure if it is worth fixing this: 1. W

[jira] [Resolved] (SPARK-18464) Spark SQL fails to load tables created without providing a schema

[ https://issues.apache.org/jira/browse/SPARK-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18464. - Resolution: Fixed Fix Version/s: 2.1.0 > Spark SQL fails to load tables created without pr

[jira] [Commented] (SPARK-18478) Support codegen for Hive UDFs

[ https://issues.apache.org/jira/browse/SPARK-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673027#comment-15673027 ] Reynold Xin commented on SPARK-18478: - Yea that it seems like it's worth doing. > S

[jira] [Commented] (SPARK-18478) Support codegen for Hive UDFs

[ https://issues.apache.org/jira/browse/SPARK-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672453#comment-15672453 ] Reynold Xin commented on SPARK-18478: - Are there any performance improvements we will

[jira] [Updated] (SPARK-16609) Single function for parsing timestamps/dates

[ https://issues.apache.org/jira/browse/SPARK-16609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16609: Assignee: (was: Reynold Xin) > Single function for parsing timestamps/dates > -

[jira] [Resolved] (SPARK-18377) warehouse path should be a static conf

[ https://issues.apache.org/jira/browse/SPARK-18377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18377. - Resolution: Fixed Fix Version/s: 2.1.0 > warehouse path should be a static conf >

[jira] [Updated] (SPARK-18300) ClassCastException during count distinct

[ https://issues.apache.org/jira/browse/SPARK-18300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18300: Fix Version/s: 2.0.3 > ClassCastException during count distinct > -

[jira] [Commented] (SPARK-18232) Support Mesos CNI

[ https://issues.apache.org/jira/browse/SPARK-18232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667797#comment-15667797 ] Reynold Xin commented on SPARK-18232: - It was only merged in master. The pr was submi

[jira] [Resolved] (SPARK-18232) Support Mesos CNI

[ https://issues.apache.org/jira/browse/SPARK-18232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18232. - Resolution: Fixed Assignee: Michael Gummelt Fix Version/s: 2.2.0 > Support Mesos

[jira] [Resolved] (SPARK-18430) Returned Message Null when Hitting an Invocation Exception of Function Lookup.

[ https://issues.apache.org/jira/browse/SPARK-18430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18430. - Resolution: Fixed Fix Version/s: 2.1.0 > Returned Message Null when Hitting an Invocation

[jira] [Resolved] (SPARK-18428) Update docs for GraphX

[ https://issues.apache.org/jira/browse/SPARK-18428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18428. - Resolution: Fixed Assignee: zhengruifeng Fix Version/s: 2.1.0 > Update docs for G

[jira] [Updated] (SPARK-18426) Python Documentation Fix for Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18426: Fix Version/s: 2.0.3 > Python Documentation Fix for Structured Streaming Programming Guide > --

[jira] [Resolved] (SPARK-18426) Python Documentation Fix for Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18426. - Resolution: Fixed Assignee: Denny Lee Fix Version/s: (was: 2.0.2)

[jira] [Updated] (SPARK-18426) Python Documentation Fix for Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18426: Fix Version/s: (was: 2.0.3) > Python Documentation Fix for Structured Streaming Programming Gui

[jira] [Resolved] (SPARK-18387) Test that expressions can be serialized

[ https://issues.apache.org/jira/browse/SPARK-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18387. - Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 2.1.0 2.0.

[jira] [Updated] (SPARK-15352) Topology aware block replication

[ https://issues.apache.org/jira/browse/SPARK-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15352: Target Version/s: 2.2.0 > Topology aware block replication > > >

[jira] [Reopened] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reopened SPARK-18367: - > DataFrame join spawns unreasonably high number of open files >

[jira] [Assigned] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reassigned SPARK-18367: --- Assignee: Reynold Xin > DataFrame join spawns unreasonably high number of open files > -

[jira] [Closed] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18367. --- Resolution: Not A Problem > DataFrame join spawns unreasonably high number of open files > --

[jira] [Closed] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18367. --- Resolution: Won't Fix > DataFrame join spawns unreasonably high number of open files > --

[jira] [Commented] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656234#comment-15656234 ] Reynold Xin commented on SPARK-18367: - Actually try set "spark.shuffle.sort.bypassMer

[jira] [Comment Edited] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656234#comment-15656234 ] Reynold Xin edited comment on SPARK-18367 at 11/11/16 5:46 AM:

[jira] [Commented] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656224#comment-15656224 ] Reynold Xin commented on SPARK-18367: - I just tried your thing and didn't see large n

[jira] [Commented] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656187#comment-15656187 ] Reynold Xin commented on SPARK-18367: - What's the number of partitions before the exc

[jira] [Resolved] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18185. - Resolution: Fixed Assignee: Eric Liang Fix Version/s: 2.1.0 > Should fix INSERT O

[jira] [Commented] (SPARK-18367) DataFrame join spawns unreasonably high number of open files

[ https://issues.apache.org/jira/browse/SPARK-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655492#comment-15655492 ] Reynold Xin commented on SPARK-18367: - Can you show the list of open files? That woul

[jira] [Updated] (SPARK-18364) expose metrics for YarnShuffleService

[ https://issues.apache.org/jira/browse/SPARK-18364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18364: Target Version/s: (was: 2.1.0) > expose metrics for YarnShuffleService >

[jira] [Updated] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

[ https://issues.apache.org/jira/browse/SPARK-18403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18403: Fix Version/s: (was: 2.1.0) 2.2.0 > ObjectHashAggregateSuite is being flaky

[jira] [Resolved] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

[ https://issues.apache.org/jira/browse/SPARK-18403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18403. - Resolution: Fixed Fix Version/s: 2.1.0 Please make sure we enable it. > ObjectHashAggreg

[jira] [Resolved] (SPARK-18302) correct several partition related behaviours of ExternalCatalog

[ https://issues.apache.org/jira/browse/SPARK-18302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18302. - Resolution: Fixed Fix Version/s: 2.1.0 > correct several partition related behaviours of E

[jira] [Resolved] (SPARK-17990) ALTER TABLE ... ADD PARTITION does not play nice with mixed-case partition column names

[ https://issues.apache.org/jira/browse/SPARK-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17990. - Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.1.0 > ALTER TABLE ... AD

[jira] [Resolved] (SPARK-17993) Spark prints an avalanche of warning messages from Parquet when reading parquet files written by older versions of Parquet-mr

[ https://issues.apache.org/jira/browse/SPARK-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17993. - Resolution: Fixed Assignee: Michael Allman Fix Version/s: 2.1.0 > Spark prints an

[jira] [Resolved] (SPARK-18262) JSON.org license is now CatX

[ https://issues.apache.org/jira/browse/SPARK-18262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18262. - Resolution: Fixed Fix Version/s: 2.1.0 > JSON.org license is now CatX > --

[jira] [Closed] (SPARK-18391) Openstack deployment scenarios

[ https://issues.apache.org/jira/browse/SPARK-18391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-18391. --- Resolution: Not A Problem > Openstack deployment scenarios > -- > >

[jira] [Resolved] (SPARK-18370) InsertIntoHadoopFsRelationCommand should keep track of its table

[ https://issues.apache.org/jira/browse/SPARK-18370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18370. - Resolution: Fixed Fix Version/s: 2.1.0 > InsertIntoHadoopFsRelationCommand should keep tra

[jira] [Updated] (SPARK-18387) Test that expressions can be serialized

[ https://issues.apache.org/jira/browse/SPARK-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18387: Target Version/s: 2.0.3, 2.1.0 (was: 2.1.0) > Test that expressions can be serialized > --

[jira] [Updated] (SPARK-18387) Test that expressions can be serialized

[ https://issues.apache.org/jira/browse/SPARK-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18387: Target Version/s: 2.1.0 Priority: Blocker (was: Major) > Test that expressions can be

[jira] [Comment Edited] (SPARK-18389) Disallow cyclic view reference

[ https://issues.apache.org/jira/browse/SPARK-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651873#comment-15651873 ] Reynold Xin edited comment on SPARK-18389 at 11/9/16 7:51 PM: -

[jira] [Commented] (SPARK-18389) Disallow cyclic view reference

[ https://issues.apache.org/jira/browse/SPARK-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651873#comment-15651873 ] Reynold Xin commented on SPARK-18389: - It'd make more sense to do this check during t

[jira] [Commented] (SPARK-18209) More robust view canonicalization without full SQL expansion

[ https://issues.apache.org/jira/browse/SPARK-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651798#comment-15651798 ] Reynold Xin commented on SPARK-18209: - Here is a ticket https://issues.apache.org/jir

[jira] [Created] (SPARK-18389) Disallow cyclic view reference

Reynold Xin created SPARK-18389: --- Summary: Disallow cyclic view reference Key: SPARK-18389 URL: https://issues.apache.org/jira/browse/SPARK-18389 Project: Spark Issue Type: Sub-task C

[jira] [Resolved] (SPARK-18368) Regular expression replace throws NullPointerException when serialized

[ https://issues.apache.org/jira/browse/SPARK-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18368. - Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 2.1.0 2.0.

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650025#comment-15650025 ] Reynold Xin commented on SPARK-18352: - Again, this has nothing to do with streaming.

[jira] [Commented] (SPARK-18350) Support session local timezone

[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649865#comment-15649865 ] Reynold Xin commented on SPARK-18350: - If it is session specific, I don't think we ne

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649835#comment-15649835 ] Reynold Xin commented on SPARK-18352: - There is already a readStream.json. "Stream"

[jira] [Updated] (SPARK-18362) Use TextFileFormat in implementation of JsonFileFormat and CSVFileFormat

[ https://issues.apache.org/jira/browse/SPARK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18362: Target Version/s: 2.2.0 > Use TextFileFormat in implementation of JsonFileFormat and CSVFileFormat

[jira] [Updated] (SPARK-18350) Support session local timezone

[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18350: Description: As of Spark 2.1, Spark SQL assumes the machine timezone for datetime manipulation, wh

[jira] [Updated] (SPARK-17703) Add unnamed version of addReferenceObj for minor objects.

[ https://issues.apache.org/jira/browse/SPARK-17703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17703: Fix Version/s: 2.0.3 > Add unnamed version of addReferenceObj for minor objects. >

[jira] [Updated] (SPARK-17924) Consolidate streaming and batch write path

[ https://issues.apache.org/jira/browse/SPARK-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17924: Target Version/s: 2.2.0 > Consolidate streaming and batch write path >

[jira] [Resolved] (SPARK-18191) Port RDD API to use commit protocol

[ https://issues.apache.org/jira/browse/SPARK-18191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18191. - Resolution: Fixed Assignee: Jiang Xingbo Fix Version/s: 2.1.0 > Port RDD API to u

[jira] [Updated] (SPARK-18219) Move commit protocol API from sql to core module

[ https://issues.apache.org/jira/browse/SPARK-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18219: Fix Version/s: (was: 2.1.0) 2.2.0 > Move commit protocol API from sql to cor

[jira] [Updated] (SPARK-16496) Add wholetext as option for reading text in SQL.

[ https://issues.apache.org/jira/browse/SPARK-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16496: Target Version/s: 2.2.0 > Add wholetext as option for reading text in SQL. > --

<    2   3   4   5   6   7   8   9   10   11   >