[jira] [Updated] (SPARK-30120) LSH approxNearestNeighbors should use TopByKeyAggregator when numNearestNeighbors is small

2019-12-03 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30120: - Description: ping [~huaxingao] > LSH approxNearestNeighbors should use TopByKeyAggregator when

[jira] [Created] (SPARK-30110) Support type judgment for ArrayData

2019-12-03 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30110: -- Summary: Support type judgment for ArrayData Key: SPARK-30110 URL: https://issues.apache.org/jira/browse/SPARK-30110 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-30101) Dataset distinct does not respect spark.default.parallelism

2019-12-03 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986722#comment-16986722 ] sam commented on SPARK-30101: - [~cloud_fan] [~kabhwan] Well this is at least a documentation error since

[jira] [Resolved] (SPARK-29537) throw exception when user defined a wrong base path

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29537. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26195

[jira] [Assigned] (SPARK-29537) throw exception when user defined a wrong base path

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29537: --- Assignee: wuyi > throw exception when user defined a wrong base path >

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-12-03 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986743#comment-16986743 ] huangtianhua commented on SPARK-29106: -- [~shaneknapp], now we don't have to install leveldbjni-all

[jira] [Created] (SPARK-30108) Add robust accumulator for observable metrics

2019-12-03 Thread Jira
Herman van Hövell created SPARK-30108: - Summary: Add robust accumulator for observable metrics Key: SPARK-30108 URL: https://issues.apache.org/jira/browse/SPARK-30108 Project: Spark

[jira] [Resolved] (SPARK-29348) Add observable metrics

2019-12-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-29348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-29348. --- Fix Version/s: 3.0.0 Resolution: Fixed > Add observable metrics >

[jira] [Reopened] (SPARK-30101) Dataset distinct does not respect spark.default.parallelism

2019-12-03 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam reopened SPARK-30101: - What is expected, is what is documented. > Dataset distinct does not respect spark.default.parallelism >

[jira] [Updated] (SPARK-30101) spark.sql.shuffle.partitions is not in Configuration docs, but a very critical parameter

2019-12-03 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam updated SPARK-30101: Summary: spark.sql.shuffle.partitions is not in Configuration docs, but a very critical parameter (was: Dataset

[jira] [Updated] (SPARK-30101) spark.sql.shuffle.partitions is not in Configuration docs, but a very critical parameter

2019-12-03 Thread sam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam updated SPARK-30101: Description: I'm creating a `SparkSession` like this: ``` SparkSession

[jira] [Commented] (SPARK-30107) Expose nested schema pruning to all V2 sources

2019-12-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986756#comment-16986756 ] Anton Okolnychyi commented on SPARK-30107: -- I'll submit a PR > Expose nested schema pruning to

[jira] [Commented] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Ruben Berenguel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986817#comment-16986817 ] Ruben Berenguel commented on SPARK-30063: - Wow, this looks bad for now (since grouped_aggs are

[jira] [Updated] (SPARK-30107) Expose nested schema pruning to all V2 sources

2019-12-03 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-30107: - Description: I think it would be great to expose the existing logic for nested schema

[jira] [Created] (SPARK-30107) Expose nested schema pruning to all V2 sources

2019-12-03 Thread Anton Okolnychyi (Jira)
Anton Okolnychyi created SPARK-30107: Summary: Expose nested schema pruning to all V2 sources Key: SPARK-30107 URL: https://issues.apache.org/jira/browse/SPARK-30107 Project: Spark Issue

[jira] [Created] (SPARK-30109) PCA use BLAS.gemv with sparse vector

2019-12-03 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-30109: Summary: PCA use BLAS.gemv with sparse vector Key: SPARK-30109 URL: https://issues.apache.org/jira/browse/SPARK-30109 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-30048) Enable aggregates with interval type values for RelationalGroupedDataset

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30048: --- Assignee: Kent Yao > Enable aggregates with interval type values for

[jira] [Resolved] (SPARK-30048) Enable aggregates with interval type values for RelationalGroupedDataset

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30048. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26681

[jira] [Commented] (SPARK-30101) spark.sql.shuffle.partitions is not in Configuration docs, but a very critical parameter

2019-12-03 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986859#comment-16986859 ] Jungtaek Lim commented on SPARK-30101: -- I'm not aware of how configuration page is constructed, but

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2019-12-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987037#comment-16987037 ] Thomas Graves commented on SPARK-18886: --- Note there is discussion on this subject on prs:

[jira] [Resolved] (SPARK-30082) Zeros are being treated as NaNs

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30082. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26738

[jira] [Assigned] (SPARK-30082) Zeros are being treated as NaNs

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30082: --- Assignee: John Ayad > Zeros are being treated as NaNs > --- >

[jira] [Resolved] (SPARK-30012) Change classes extending scala collection classes to work with 2.13

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30012. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26728

[jira] [Assigned] (SPARK-30012) Change classes extending scala collection classes to work with 2.13

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30012: - Assignee: Sean R. Owen > Change classes extending scala collection classes to work

[jira] [Resolved] (SPARK-30083) visitArithmeticUnary should wrap PLUS case with UnaryPositive for type checking

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30083. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26716

[jira] [Assigned] (SPARK-30083) visitArithmeticUnary should wrap PLUS case with UnaryPositive for type checking

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30083: --- Assignee: Kent Yao > visitArithmeticUnary should wrap PLUS case with UnaryPositive for

[jira] [Updated] (SPARK-29477) Improve tooltip information for Streaming Tab

2019-12-03 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29477: - Priority: Minor (was: Major) > Improve tooltip information for Streaming Tab >

[jira] [Assigned] (SPARK-29477) Improve tooltip information for Streaming Tab

2019-12-03 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29477: Assignee: Rakesh Raushan > Improve tooltip information for Streaming Tab >

[jira] [Resolved] (SPARK-29477) Improve tooltip information for Streaming Tab

2019-12-03 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29477. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26467

[jira] [Commented] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987154#comment-16987154 ] Bryan Cutler commented on SPARK-30063: -- I haven't looked at your bug report in detail but you are

[jira] [Commented] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Tim Kellogg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987230#comment-16987230 ] Tim Kellogg commented on SPARK-30063: - Improvement suggestions * Set the PYTHONHASHSEED environment

[jira] [Commented] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987120#comment-16987120 ] Aman Omer commented on SPARK-29667: --- Actually there is also JIRA for decimal type mismatching due to

[jira] [Commented] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987119#comment-16987119 ] Aman Omer commented on SPARK-29667: --- Hi [~jessielin], did you observed same issue with any other data

[jira] [Created] (SPARK-30111) spark R dockerfile fails to build

2019-12-03 Thread Shane Knapp (Jira)
Shane Knapp created SPARK-30111: --- Summary: spark R dockerfile fails to build Key: SPARK-30111 URL: https://issues.apache.org/jira/browse/SPARK-30111 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-12-03 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987143#comment-16987143 ] Shane Knapp commented on SPARK-29106: - got it, thanks! > Add jenkins arm test for spark >

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2019-12-03 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987148#comment-16987148 ] Shane Knapp commented on SPARK-29988: - i probably won't get around to this until next week... >

[jira] [Comment Edited] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987120#comment-16987120 ] Aman Omer edited comment on SPARK-29667 at 12/3/19 7:03 PM: Actually there

[jira] [Resolved] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30060. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26692

[jira] [Assigned] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30060: - Assignee: Luca Canali > Uniform naming for Spark Metrics configuration parameters >

[jira] [Commented] (SPARK-29748) Remove sorting of fields in PySpark SQL Row creation

2019-12-03 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987252#comment-16987252 ] Bryan Cutler commented on SPARK-29748: -- [~zero323] I made some updates to the PR with remove the

[jira] [Assigned] (SPARK-30106) DynamicPartitionPruningSuite#"no predicate on the dimension table" is not be tested

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30106: - Assignee: deshanxiao > DynamicPartitionPruningSuite#"no predicate on the dimension

[jira] [Resolved] (SPARK-30106) DynamicPartitionPruningSuite#"no predicate on the dimension table" is not be tested

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30106. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26744

[jira] [Resolved] (SPARK-30051) Clean up hadoop-3.2 transitive dependencies

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30051. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26742

[jira] [Assigned] (SPARK-30051) Clean up hadoop-3.2 transitive dependencies

2019-12-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30051: - Assignee: Dongjoon Hyun > Clean up hadoop-3.2 transitive dependencies >

[jira] [Commented] (SPARK-30111) spark R dockerfile fails to build

2019-12-03 Thread Ilan Filonenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987372#comment-16987372 ] Ilan Filonenko commented on SPARK-30111: The error seems to be from: {code:yaml} Step 6/12 :

[jira] [Created] (SPARK-30112) Insert overwrite should be able to overwrite to same table under dynamic partition overwrite

2019-12-03 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-30112: --- Summary: Insert overwrite should be able to overwrite to same table under dynamic partition overwrite Key: SPARK-30112 URL: https://issues.apache.org/jira/browse/SPARK-30112

[jira] [Resolved] (SPARK-29903) Add documentation for recursiveFileLookup

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29903. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed in

[jira] [Commented] (SPARK-29988) Adjust Jenkins jobs for `hive-1.2/2.3` combination

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987413#comment-16987413 ] Hyukjin Kwon commented on SPARK-29988: -- Sure, take your time. just as a reminder, Hive version can

[jira] [Created] (SPARK-30113) Document mergeSchema option in Python Orc APIs

2019-12-03 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-30113: Summary: Document mergeSchema option in Python Orc APIs Key: SPARK-30113 URL: https://issues.apache.org/jira/browse/SPARK-30113 Project: Spark Issue

[jira] [Commented] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987417#comment-16987417 ] Hyukjin Kwon commented on SPARK-30063: -- {quote} Set the PYTHONHASHSEED environment variable such

[jira] [Created] (SPARK-30117) Improve limit only query on Hive table and view

2019-12-03 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-30117: -- Summary: Improve limit only query on Hive table and view Key: SPARK-30117 URL: https://issues.apache.org/jira/browse/SPARK-30117 Project: Spark Issue Type:

[jira] [Created] (SPARK-30116) Improve limit only query on views

2019-12-03 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-30116: -- Summary: Improve limit only query on views Key: SPARK-30116 URL: https://issues.apache.org/jira/browse/SPARK-30116 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-30109) PCA use BLAS.gemv with sparse vector

2019-12-03 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30109: Assignee: zhengruifeng > PCA use BLAS.gemv with sparse vector >

[jira] [Resolved] (SPARK-30109) PCA use BLAS.gemv with sparse vector

2019-12-03 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30109. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26745

[jira] [Commented] (SPARK-30062) bug with DB2Driver using mode("overwrite") option("truncate",True)

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987424#comment-16987424 ] Hyukjin Kwon commented on SPARK-30062: -- Can you open a PR to fix DB2 dialect then? > bug with

[jira] [Resolved] (SPARK-30110) Support type judgment for ArrayData

2019-12-03 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng resolved SPARK-30110. Resolution: Won't Fix > Support type judgment for ArrayData > ---

[jira] [Commented] (SPARK-30062) bug with DB2Driver using mode("overwrite") option("truncate",True)

2019-12-03 Thread Guy Huinen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987432#comment-16987432 ] Guy Huinen commented on SPARK-30062: ok > bug with DB2Driver using mode("overwrite")

[jira] [Resolved] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29667. -- Resolution: Duplicate I am resolving this as a duplicate of SPARK-29860. > implicitly

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-12-03 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987407#comment-16987407 ] huangtianhua commented on SPARK-29106: -- [~shaneknapp], ok, thanks, sorry and I found the mvn

[jira] [Commented] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987418#comment-16987418 ] Hyukjin Kwon commented on SPARK-30063: -- Closing this assuming the issue was resolved. > Failure

[jira] [Resolved] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30063. -- Resolution: Not A Problem > Failure when returning a value from multiple Pandas UDFs >

[jira] [Updated] (SPARK-30091) Document mergeSchema option directly in the Python Parquet APIs

2019-12-03 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30091: - Summary: Document mergeSchema option directly in the Python Parquet APIs (was:

[jira] [Updated] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29667: - Description: Ran into error on this sql Mismatched columns: {code} [(a.`id`:decimal(28,0),

[jira] [Created] (SPARK-30114) Optimize LIMIT only query by partial listing files

2019-12-03 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-30114: -- Summary: Optimize LIMIT only query by partial listing files Key: SPARK-30114 URL: https://issues.apache.org/jira/browse/SPARK-30114 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30111) spark R dockerfile fails to build

2019-12-03 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-30111. - Resolution: Fixed > spark R dockerfile fails to build > - > >

[jira] [Created] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
John Zhuge created SPARK-30118: -- Summary: ALTER VIEW QUERY does not work Key: SPARK-30118 URL: https://issues.apache.org/jira/browse/SPARK-30118 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-30111) spark R dockerfile fails to build

2019-12-03 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp reassigned SPARK-30111: --- Assignee: Ilan Filonenko > spark R dockerfile fails to build >

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-12-03 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987435#comment-16987435 ] Shane Knapp commented on SPARK-29106: - nice catch...  fixed and relaunched. > Add jenkins arm test

[jira] [Commented] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987440#comment-16987440 ] Hyukjin Kwon commented on SPARK-29667: -- I think this issue is a duplicate of SPARK-29860. [The PR

[jira] [Assigned] (SPARK-29903) Add documentation for recursiveFileLookup

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29903: Assignee: Nicholas Chammas > Add documentation for recursiveFileLookup >

[jira] [Updated] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29667: - Description: Ran into error on this sql Mismatched columns: {code} [(a.`id`:decimal(28,0),

[jira] [Updated] (SPARK-30114) Optimize LIMIT only query by partial listing files

2019-12-03 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-30114: --- Description: We use Spark as ad-hoc query engine. Most of users' SELECT queries with LIMIT

[jira] [Updated] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29600: - Description: {code} SELECT array_contains(array(0,0.1,0.2,0.3,0.5,0.02,0.033), .2); {code}

[jira] [Updated] (SPARK-30114) Improve LIMIT only query by partial listing files

2019-12-03 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-30114: --- Summary: Improve LIMIT only query by partial listing files (was: Optimize LIMIT only query by

[jira] [Created] (SPARK-30115) Improve limit only query on datasource table

2019-12-03 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-30115: -- Summary: Improve limit only query on datasource table Key: SPARK-30115 URL: https://issues.apache.org/jira/browse/SPARK-30115 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29591) Support data insertion in a different order if you wish or even omit some columns in spark sql also like postgresql

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29591: - Description: Support data insertion in a different order if you wish or even omit some columns

[jira] [Updated] (SPARK-29591) Support data insertion in a different order if you wish or even omit some columns in spark sql also like postgresql

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29591: - Description: Support data insertion in a different order if you wish or even omit some columns

[jira] [Updated] (SPARK-29591) Support data insertion in a different order if you wish or even omit some columns in spark sql also like postgresql

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29591: - Description: Support data insertion in a different order if you wish or even omit some columns

[jira] [Resolved] (SPARK-27025) Speed up toLocalIterator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27025. -- Resolution: Duplicate Actually, I think this was done by SPARK-27659 .. > Speed up

[jira] [Reopened] (SPARK-27025) Speed up toLocalIterator

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27025: -- > Speed up toLocalIterator > > > Key: SPARK-27025 >

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987428#comment-16987428 ] John Zhuge commented on SPARK-30118: {code:java} spark-sql> DESC FORMATTED jzhuge.v1; foo1string

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987429#comment-16987429 ] John Zhuge commented on SPARK-30118: I am running Spark master with Hive 1.2.1. Same issue in Spark

[jira] [Commented] (SPARK-27547) fix DataFrame self-join problems

2019-12-03 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987444#comment-16987444 ] Nicholas Chammas commented on SPARK-27547: -- Should this be marked as resolved by

[jira] [Assigned] (SPARK-30091) Document mergeSchema option directly in the Python Parquet APIs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30091: Assignee: Nicholas Chammas > Document mergeSchema option directly in the Python Parquet

[jira] [Resolved] (SPARK-30091) Document mergeSchema option directly in the Python Parquet APIs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30091. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26730

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-12-03 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987456#comment-16987456 ] huangtianhua commented on SPARK-29106: -- [~shaneknapp],:) So seems you're back:)  Could you please

[jira] [Resolved] (SPARK-30113) Document mergeSchema option in Python Orc APIs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30113. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26755

[jira] [Assigned] (SPARK-30113) Document mergeSchema option in Python Orc APIs

2019-12-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30113: Assignee: Nicholas Chammas > Document mergeSchema option in Python Orc APIs >

[jira] [Comment Edited] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987491#comment-16987491 ] Lantao Jin edited comment on SPARK-30118 at 12/4/19 3:42 AM: - I think it had

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987491#comment-16987491 ] Lantao Jin commented on SPARK-30118: I think it had been fixed in latest version. Could you try it

[jira] [Created] (SPARK-30119) Support pagination for spark streaming tab

2019-12-03 Thread jobit mathew (Jira)
jobit mathew created SPARK-30119: Summary: Support pagination for spark streaming tab Key: SPARK-30119 URL: https://issues.apache.org/jira/browse/SPARK-30119 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-27547) fix DataFrame self-join problems

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27547. - Resolution: Duplicate > fix DataFrame self-join problems > > >

[jira] [Commented] (SPARK-30119) Support pagination for spark streaming tab

2019-12-03 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987522#comment-16987522 ] Rakesh Raushan commented on SPARK-30119: I will raise PR for this one. > Support pagination for

[jira] [Updated] (SPARK-30119) Support pagination for spark streaming tab

2019-12-03 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-30119: - Affects Version/s: 3.0.0 > Support pagination for spark streaming tab >

[jira] [Updated] (SPARK-30082) Zeros are being treated as NaNs

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30082: Fix Version/s: 2.4.5 > Zeros are being treated as NaNs > --- > >

[jira] [Assigned] (SPARK-30099) Improve Analyzed Logical Plan as duplicate AnalysisExceptions are coming

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30099: --- Assignee: jobit mathew > Improve Analyzed Logical Plan as duplicate AnalysisExceptions are

[jira] [Resolved] (SPARK-30099) Improve Analyzed Logical Plan as duplicate AnalysisExceptions are coming

2019-12-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30099. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26734

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-12-03 Thread jugosag (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987593#comment-16987593 ] jugosag commented on SPARK-28921: - We are also observing this on two of our clusters, both set up with

[jira] [Created] (SPARK-30120) LSH approxNearestNeighbors should use TopByKeyAggregator when numNearestNeighbors is small

2019-12-03 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-30120: Summary: LSH approxNearestNeighbors should use TopByKeyAggregator when numNearestNeighbors is small Key: SPARK-30120 URL: https://issues.apache.org/jira/browse/SPARK-30120