[jira] [Commented] (SPARK-23911) High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R

[ https://issues.apache.org/jira/browse/SPARK-23911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569108#comment-16569108 ] Xiao Li commented on SPARK-23911: - Sounds good to me. Just changed reduce to aggregate

[jira] [Created] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

Xiao Li created SPARK-25025: --- Summary: Remove the default value of isAll in INTERSECT/EXCEPT Key: SPARK-25025 URL: https://issues.apache.org/jira/browse/SPARK-25025 Project: Spark Issue Type: Impro

[jira] [Resolved] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

[ https://issues.apache.org/jira/browse/SPARK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25025. - Resolution: Fixed Fix Version/s: 2.4.0 > Remove the default value of isAll in INTERSECT/EXCEPT >

[jira] [Assigned] (SPARK-24996) Use DSL to simplify DeclarativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-24996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-24996: --- Assignee: Marco Gaido > Use DSL to simplify DeclarativeAggregate >

[jira] [Resolved] (SPARK-24996) Use DSL to simplify DeclarativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-24996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24996. - Resolution: Fixed Fix Version/s: 2.4.0 > Use DSL to simplify DeclarativeAggregate > -

[jira] [Updated] (SPARK-25031) The schema of MapType can not be printed correctly

[ https://issues.apache.org/jira/browse/SPARK-25031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25031: Target Version/s: 2.4.0 > The schema of MapType can not be printed correctly > ---

[jira] [Resolved] (SPARK-24979) add AnalysisHelper#resolveOperatorsUp

[ https://issues.apache.org/jira/browse/SPARK-24979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24979. - Resolution: Fixed Fix Version/s: 2.4.0 > add AnalysisHelper#resolveOperatorsUp >

[jira] [Assigned] (SPARK-25046) Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO"

[ https://issues.apache.org/jira/browse/SPARK-25046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25046: --- Assignee: SongXun > Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO" > ---

[jira] [Resolved] (SPARK-25046) Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO"

[ https://issues.apache.org/jira/browse/SPARK-25046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25046. - Resolution: Fixed Fix Version/s: 2.4.0 > Alter View can excute sql like "ALTER VIEW ... AS INSE

[jira] [Resolved] (SPARK-25063) Rename class KnowNotNull to KnownNotNull

[ https://issues.apache.org/jira/browse/SPARK-25063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25063. - Resolution: Fixed Assignee: Maryann Xue Fix Version/s: 2.4.0 > Rename class KnowNotNull

[jira] [Resolved] (SPARK-25077) Delete unused variable in WindowExec

[ https://issues.apache.org/jira/browse/SPARK-25077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25077. - Resolution: Fixed Assignee: Li Yuanjian Fix Version/s: 2.4.0 > Delete unused variable in

[jira] [Resolved] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25076. - Resolution: Fixed Fix Version/s: 2.4.0 > SQLConf should not be retrieved from a stopped SparkSess

[jira] [Resolved] (SPARK-25068) High-order function: exists(array, function) → boolean

[ https://issues.apache.org/jira/browse/SPARK-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25068. - Resolution: Fixed Fix Version/s: 2.4.0 > High-order function: exists(array, function) → boolean >

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575790#comment-16575790 ] Xiao Li commented on SPARK-25084: - Let us mark it as a blocker. How about the master bra

[jira] [Updated] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25084: Priority: Blocker (was: Major) > "distribute by" on multiple columns may lead to codegen issue >

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575791#comment-16575791 ] Xiao Li commented on SPARK-25084: - Could you investigate which PR introduced this bug? W

[jira] [Resolved] (SPARK-25092) Add RewriteExceptAll and RewriteIntersectAll in the list of nonExcludableRules

[ https://issues.apache.org/jira/browse/SPARK-25092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25092. - Resolution: Fixed Assignee: Dilip Biswal Fix Version/s: 2.4.0 > Add RewriteExceptAll and

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25081: Fix Version/s: 2.2.3 > Nested spill in ShuffleExternalSorter may access a released memory page >

[jira] [Commented] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

[ https://issues.apache.org/jira/browse/SPARK-24156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578937#comment-16578937 ] Xiao Li commented on SPARK-24156: - [~tdas] Can we mark it done? > Enable no-data micro

[jira] [Commented] (SPARK-25051) where clause on dataset gives AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-25051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579967#comment-16579967 ] Xiao Li commented on SPARK-25051: - [~mgaido] This breaks the backport rule. We are unabl

[jira] [Updated] (SPARK-25051) where clause on dataset gives AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-25051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25051: Component/s: (was: Spark Core) > where clause on dataset gives AnalysisException > ---

[jira] [Updated] (SPARK-25051) where clause on dataset gives AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-25051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25051: Fix Version/s: (was: 2.3.3) 2.3.2 > where clause on dataset gives AnalysisException

[jira] [Resolved] (SPARK-25051) where clause on dataset gives AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-25051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25051. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.3.3 > where clause on dataset g

[jira] [Commented] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580153#comment-16580153 ] Xiao Li commented on SPARK-25114: - [~jerryshao] Another blocker for 2.3 > RecordBinaryC

[jira] [Updated] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25114: Target Version/s: 2.3.2, 2.4.0 (was: 2.4.0) > RecordBinaryComparator may return wrong result when subtrac

[jira] [Commented] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580298#comment-16580298 ] Xiao Li commented on SPARK-24838: - [~maurits] Any update? Also cc [~liwen] [~hvanhovel

[jira] [Updated] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24838: Labels: (was: spree) > Support uncorrelated IN/EXISTS subqueries for more operators > -

[jira] [Updated] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24838: Target Version/s: 2.4.0 > Support uncorrelated IN/EXISTS subqueries for more operators >

[jira] [Resolved] (SPARK-25113) Add logging to CodeGenerator when any generated method's bytecode size goes above HugeMethodLimit

[ https://issues.apache.org/jira/browse/SPARK-25113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25113. - Resolution: Fixed Assignee: Kris Mok Fix Version/s: 2.4.0 > Add logging to CodeGenerator

[jira] [Created] (SPARK-25121) Support multi-part column name for hint resolution

Xiao Li created SPARK-25121: --- Summary: Support multi-part column name for hint resolution Key: SPARK-25121 URL: https://issues.apache.org/jira/browse/SPARK-25121 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-25121) Support multi-part column name for hint resolution

[ https://issues.apache.org/jira/browse/SPARK-25121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580779#comment-16580779 ] Xiao Li commented on SPARK-25121: - cc [~ksunitha] [~dkbiswal] > Support multi-part colu

[jira] [Updated] (SPARK-25088) Rest Server default & doc updates

[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25088: Labels: release-notes (was: ) > Rest Server default & doc updates > - > >

[jira] [Resolved] (SPARK-25031) The schema of MapType can not be printed correctly

[ https://issues.apache.org/jira/browse/SPARK-25031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25031. - Resolution: Fixed Assignee: Hao Ren Fix Version/s: 2.4.0 > The schema of MapType can not

[jira] [Commented] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583171#comment-16583171 ] Xiao Li commented on SPARK-24838: - [~dkbiswal] Can you submit a fix? > Support uncorrel

[jira] [Updated] (SPARK-25135) insert datasource table may all null when select from view

[ https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25135: Labels: correctness (was: ) > insert datasource table may all null when select from view > --

[jira] [Updated] (SPARK-25135) insert datasource table may all null when select from view

[ https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25135: Priority: Blocker (was: Major) > insert datasource table may all null when select from view > ---

[jira] [Updated] (SPARK-25135) insert datasource table may all null when select from view

[ https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25135: Target Version/s: 2.3.2 > insert datasource table may all null when select from view > ---

[jira] [Updated] (SPARK-25138) Spark Shell should show the Scala prompt after initialization is complete

[ https://issues.apache.org/jira/browse/SPARK-25138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25138: Target Version/s: 2.4.0 > Spark Shell should show the Scala prompt after initialization is complete >

[jira] [Commented] (SPARK-25138) Spark Shell should show the Scala prompt after initialization is complete

[ https://issues.apache.org/jira/browse/SPARK-25138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583435#comment-16583435 ] Xiao Li commented on SPARK-25138: - This is a regression. cc [~mgaido] > Spark Shell sho

[jira] [Commented] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583436#comment-16583436 ] Xiao Li commented on SPARK-25140: - cc [~maropu] > Add optional logging to UnsafeProject

[jira] [Updated] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25140: Issue Type: Improvement (was: Wish) > Add optional logging to UnsafeProjection.create when it falls back

[jira] [Updated] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25140: Target Version/s: 2.4.0 > Add optional logging to UnsafeProjection.create when it falls back to > interpr

[jira] [Commented] (SPARK-25138) Spark Shell should show the Scala prompt after initialization is complete

[ https://issues.apache.org/jira/browse/SPARK-25138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584072#comment-16584072 ] Xiao Li commented on SPARK-25138: - [~mgaido] Thanks! > Spark Shell should show the Sca

[jira] [Updated] (SPARK-24959) Do not invoke the CSV/JSON parser for empty schema

[ https://issues.apache.org/jira/browse/SPARK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24959: Priority: Major (was: Minor) > Do not invoke the CSV/JSON parser for empty schema > -

[jira] [Resolved] (SPARK-24959) Do not invoke the CSV/JSON parser for empty schema

[ https://issues.apache.org/jira/browse/SPARK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24959. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Do not invoke the CSV/JSON

[jira] [Resolved] (SPARK-23573) Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in SQL modules

[ https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23573. - Resolution: Duplicate Assignee: Gengliang Wang Fix Version/s: 2.4.0 > Create linter rule

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586593#comment-16586593 ] Xiao Li commented on SPARK-23874: - [~bryanc]To get these fixes, we need to upgrade pyarr

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587702#comment-16587702 ] Xiao Li commented on SPARK-23874: - Great! Then, users can keep using pyarrow 0.8 and get

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587841#comment-16587841 ] Xiao Li commented on SPARK-25178: - cc [~kiszk] Do you have a bandwidth to take this? >

[jira] [Created] (SPARK-25179) Document the features that require Pyarrow 0.10

Xiao Li created SPARK-25179: --- Summary: Document the features that require Pyarrow 0.10 Key: SPARK-25179 URL: https://issues.apache.org/jira/browse/SPARK-25179 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25179: Issue Type: Documentation (was: Improvement) > Document the features that require Pyarrow 0.10 >

[jira] [Commented] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587847#comment-16587847 ] Xiao Li commented on SPARK-25179: - cc [~bryanc] > Document the features that require Py

[jira] [Updated] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25179: Description: binary type support requires pyarrow 0.10.0. (was: binary type support requires pyarrow 0.1

[jira] [Commented] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587884#comment-16587884 ] Xiao Li commented on SPARK-25179: - Thanks! It is not urgent as long as it is documented

[jira] [Assigned] (SPARK-25179) Document the features that require Pyarrow 0.10

[ https://issues.apache.org/jira/browse/SPARK-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25179: --- Assignee: Bryan Cutler > Document the features that require Pyarrow 0.10 >

[jira] [Updated] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24307: Labels: releasenotes (was: ) > Support sending messages over 2GB from memory > --

[jira] [Updated] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24307: Labels: release-notes (was: releasenotes) > Support sending messages over 2GB from memory > -

[jira] [Resolved] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25114. - Resolution: Fixed Assignee: Jiang Xingbo Fix Version/s: 2.4.0 2.3.2 >

[jira] [Commented] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588121#comment-16588121 ] Xiao Li commented on SPARK-25114: - Let us update the fix version after the fix of 2.2 is

[jira] [Resolved] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25140. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.4.0 > Add optional logging

[jira] [Resolved] (SPARK-25159) json schema inference should only trigger one job

[ https://issues.apache.org/jira/browse/SPARK-25159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25159. - Resolution: Fixed Target Version/s: 2.4.0 > json schema inference should only trigger one job

[jira] [Updated] (SPARK-25159) json schema inference should only trigger one job

[ https://issues.apache.org/jira/browse/SPARK-25159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25159: Fix Version/s: 2.4.0 > json schema inference should only trigger one job > ---

[jira] [Resolved] (SPARK-24882) data source v2 API improvement

[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24882. - Resolution: Fixed Fix Version/s: 2.4.0 > data source v2 API improvement > ---

[jira] [Commented] (SPARK-24882) data source v2 API improvement

[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588462#comment-16588462 ] Xiao Li commented on SPARK-24882: - Please continue the discussions in the https://issue

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589014#comment-16589014 ] Xiao Li commented on SPARK-25196: - cc [~rxin] > Analyze column statistics in cached que

[jira] [Commented] (SPARK-25196) Analyze column statistics in cached query

[ https://issues.apache.org/jira/browse/SPARK-25196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589013#comment-16589013 ] Xiao Li commented on SPARK-25196: - This sounds reasonable to me. > Analyze column stat

[jira] [Commented] (SPARK-25178) Use dummy name for xxxHashMapGenerator key/value schema field

[ https://issues.apache.org/jira/browse/SPARK-25178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589015#comment-16589015 ] Xiao Li commented on SPARK-25178: - Thanks! > Use dummy name for xxxHashMapGenerator key

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589018#comment-16589018 ] Xiao Li commented on SPARK-23874: - [~bryanc] Could you list some examples that can affec

[jira] [Updated] (SPARK-25114) RecordBinaryComparator may return wrong result when subtraction between two words is divisible by Integer.MAX_VALUE

[ https://issues.apache.org/jira/browse/SPARK-25114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25114: Fix Version/s: 2.2.3 > RecordBinaryComparator may return wrong result when subtraction between two > word

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23207: Fix Version/s: 2.2.3 > Shuffle+Repartition on an DataFrame could lead to incorrect answers > -

[jira] [Updated] (SPARK-24564) Add test suite for RecordBinaryComparator

[ https://issues.apache.org/jira/browse/SPARK-24564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24564: Fix Version/s: 2.3.2 2.2.3 > Add test suite for RecordBinaryComparator > --

[jira] [Commented] (SPARK-24814) Relationship between catalog and datasources

[ https://issues.apache.org/jira/browse/SPARK-24814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591029#comment-16591029 ] Xiao Li commented on SPARK-24814: - Also cc [~cloud_fan] > Relationship between catalog

[jira] [Assigned] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-4502: -- Assignee: Michael Allman > Spark SQL reads unneccesary nested fields from Parquet > --

[jira] [Resolved] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-4502. Resolution: Fixed Fix Version/s: 2.4.0 > Spark SQL reads unneccesary nested fields from Parquet > --

[jira] [Commented] (SPARK-25206) Wrong data may be returned for Parquet

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593041#comment-16593041 ] Xiao Li commented on SPARK-25206: - Previously, we do not have a good test coverage when

[jira] [Comment Edited] (SPARK-25206) Wrong data may be returned for Parquet

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593041#comment-16593041 ] Xiao Li edited comment on SPARK-25206 at 8/26/18 10:45 PM: --- Cu

[jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593840#comment-16593840 ] Xiao Li commented on SPARK-25206: - Silently ignoring it is bad. We should issue an excep

[jira] [Updated] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25206: Target Version/s: 2.3.2, 2.4.0 (was: 2.3.2) > wrong records are returned when Hive metastore schema and p

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593913#comment-16593913 ] Xiao Li commented on SPARK-23874: - ping [~bryanc] again. > Upgrade apache/arrow to 0.1

[jira] [Updated] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25240: Priority: Blocker (was: Major) > A deadlock in ALTER TABLE RECOVER PARTITIONS > -

[jira] [Updated] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25240: Target Version/s: 2.4.0 > A deadlock in ALTER TABLE RECOVER PARTITIONS > -

[jira] [Updated] (SPARK-25164) Parquet reader builds entire list of columns once for each column

[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25164: Fix Version/s: 2.3.2 2.2.3 > Parquet reader builds entire list of columns once for each

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23207: Fix Version/s: 2.1.4 > Shuffle+Repartition on an DataFrame could lead to incorrect answers > -

[jira] [Resolved] (SPARK-23997) Configurable max number of buckets

[ https://issues.apache.org/jira/browse/SPARK-23997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23997. - Resolution: Fixed Assignee: Fernando Pereira Fix Version/s: 2.4.0 > Configurable max num

[jira] [Created] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

Xiao Li created SPARK-25267: --- Summary: Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive Key: SPARK-25267 URL: https://issues.apache.org/jira/browse/SPARK-25267 Project: Spark

[jira] [Commented] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

[ https://issues.apache.org/jira/browse/SPARK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595388#comment-16595388 ] Xiao Li commented on SPARK-25267: - cc [~dkbiswal] > Disable ConvertToLocalRelation in t

[jira] [Resolved] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25240. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > A deadlock in ALTER TABLE

[jira] [Resolved] (SPARK-25212) Support Filter in ConvertToLocalRelation

[ https://issues.apache.org/jira/browse/SPARK-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25212. - Resolution: Fixed Assignee: Bogdan Raducanu Fix Version/s: 2.4.0 > Support Filter in Con

[jira] [Updated] (SPARK-24005) Remove usage of Scala’s parallel collection

[ https://issues.apache.org/jira/browse/SPARK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24005: Labels: (was: starter) > Remove usage of Scala’s parallel collection > -

[jira] [Updated] (SPARK-25286) Remove dangerous parmap

[ https://issues.apache.org/jira/browse/SPARK-25286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25286: Target Version/s: 2.4.0 > Remove dangerous parmap > --- > > Key: SPARK

[jira] [Created] (SPARK-25296) Create ExplainSuite

Xiao Li created SPARK-25296: --- Summary: Create ExplainSuite Key: SPARK-25296 URL: https://issues.apache.org/jira/browse/SPARK-25296 Project: Spark Issue Type: Test Components: SQL Affe

[jira] [Resolved] (SPARK-25296) Create ExplainSuite

[ https://issues.apache.org/jira/browse/SPARK-25296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25296. - Resolution: Fixed Fix Version/s: 2.4.0 > Create ExplainSuite > --- > >

[jira] [Resolved] (SPARK-25286) Remove dangerous parmap

[ https://issues.apache.org/jira/browse/SPARK-25286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25286. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Remove dangerous parmap >

[jira] [Created] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

Xiao Li created SPARK-25312: --- Summary: Add description for the conf spark.network.crypto.keyFactoryIterations Key: SPARK-25312 URL: https://issues.apache.org/jira/browse/SPARK-25312 Project: Spark

[jira] [Updated] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25312: Labels: starter (was: ) > Add description for the conf spark.network.crypto.keyFactoryIterations > --

[jira] [Updated] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22357: Labels: behavior-changes (was: ) > SparkContext.binaryFiles ignore minPartitions parameter >

[jira] [Created] (SPARK-25328) Add a test case for having two columns as the grouping key in group aggregate pandas UDF

Xiao Li created SPARK-25328: --- Summary: Add a test case for having two columns as the grouping key in group aggregate pandas UDF Key: SPARK-25328 URL: https://issues.apache.org/jira/browse/SPARK-25328 Projec

[jira] [Commented] (SPARK-25328) Add a test case for having two columns as the grouping key in group aggregate pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602630#comment-16602630 ] Xiao Li commented on SPARK-25328: - cc [~icexelloss] [~bryanc] [~hyukjin.kwon] > Add a t

[jira] [Updated] (SPARK-25328) Add an example for having two columns as the grouping key in group aggregate pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25328: Summary: Add an example for having two columns as the grouping key in group aggregate pandas UDF (was: Ad

[jira] [Commented] (SPARK-25328) Add a test case for having two columns as the grouping key in group aggregate pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602643#comment-16602643 ] Xiao Li commented on SPARK-25328: - Uh, I see. Could we add an example in functions.py? I

<    4   5   6   7   8   9   10   11   12   13   >