[jira] [Updated] (SPARK-25376) Scenarios we should handle but missed for barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-25376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25376: -- Description: [~irashid] You mentioned that there are couple scenarios we should handle in

[jira] [Created] (SPARK-25376) Scenarios we should handle but missed for barrier execution mode

Xiangrui Meng created SPARK-25376: - Summary: Scenarios we should handle but missed for barrier execution mode Key: SPARK-25376 URL: https://issues.apache.org/jira/browse/SPARK-25376 Project: Spark

[jira] [Resolved] (SPARK-25225) Add support for "List"-Type columns

[ https://issues.apache.org/jira/browse/SPARK-25225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25225. -- Resolution: Won't Fix Let me leave this resolve for now. I don't think we will add this type.

[jira] [Assigned] (SPARK-25375) Reenable qualified perm. function checks in UDFSuite

[ https://issues.apache.org/jira/browse/SPARK-25375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25375: Assignee: (was: Apache Spark) > Reenable qualified perm. function checks in UDFSuite

[jira] [Assigned] (SPARK-25375) Reenable qualified perm. function checks in UDFSuite

[ https://issues.apache.org/jira/browse/SPARK-25375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25375: Assignee: Apache Spark > Reenable qualified perm. function checks in UDFSuite >

[jira] [Commented] (SPARK-25375) Reenable qualified perm. function checks in UDFSuite

[ https://issues.apache.org/jira/browse/SPARK-25375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607842#comment-16607842 ] Apache Spark commented on SPARK-25375: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-25375) Reenable qualified perm. function checks in UDFSuite

Dongjoon Hyun created SPARK-25375: - Summary: Reenable qualified perm. function checks in UDFSuite Key: SPARK-25375 URL: https://issues.apache.org/jira/browse/SPARK-25375 Project: Spark Issue

[jira] [Updated] (SPARK-25374) SafeProjection supports fallback to an interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25374: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-23580 > SafeProjection

[jira] [Commented] (SPARK-25374) SafeProjection supports fallback to an interpreted mode

[ https://issues.apache.org/jira/browse/SPARK-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607837#comment-16607837 ] Takeshi Yamamuro commented on SPARK-25374: -- WIP:

[jira] [Created] (SPARK-25374) SafeProjection supports fallback to an interpreted mode

Takeshi Yamamuro created SPARK-25374: Summary: SafeProjection supports fallback to an interpreted mode Key: SPARK-25374 URL: https://issues.apache.org/jira/browse/SPARK-25374 Project: Spark

[jira] [Created] (SPARK-25373) Support mixed language pipelines on Spark on K8s

holdenk created SPARK-25373: --- Summary: Support mixed language pipelines on Spark on K8s Key: SPARK-25373 URL: https://issues.apache.org/jira/browse/SPARK-25373 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25372) Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit

[ https://issues.apache.org/jira/browse/SPARK-25372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25372: Assignee: (was: Apache Spark) > Deprecate Yarn-specific configs in regards to keytab

[jira] [Assigned] (SPARK-25372) Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit

[ https://issues.apache.org/jira/browse/SPARK-25372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25372: Assignee: Apache Spark > Deprecate Yarn-specific configs in regards to keytab login for

[jira] [Commented] (SPARK-25372) Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit

[ https://issues.apache.org/jira/browse/SPARK-25372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607739#comment-16607739 ] Apache Spark commented on SPARK-25372: -- User 'ifilonenko' has created a pull request for this

[jira] [Commented] (SPARK-25372) Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit

[ https://issues.apache.org/jira/browse/SPARK-25372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607738#comment-16607738 ] Apache Spark commented on SPARK-25372: -- User 'ifilonenko' has created a pull request for this

[jira] [Created] (SPARK-25372) Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit

Ilan Filonenko created SPARK-25372: -- Summary: Deprecate Yarn-specific configs in regards to keytab login for SparkSubmit Key: SPARK-25372 URL: https://issues.apache.org/jira/browse/SPARK-25372

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607543#comment-16607543 ] Apache Spark commented on SPARK-25317: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607539#comment-16607539 ] Apache Spark commented on SPARK-23762: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607540#comment-16607540 ] Apache Spark commented on SPARK-23762: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607537#comment-16607537 ] Apache Spark commented on SPARK-23762: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

[ https://issues.apache.org/jira/browse/SPARK-23879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607536#comment-16607536 ] Apache Spark commented on SPARK-23879: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

[ https://issues.apache.org/jira/browse/SPARK-23879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607535#comment-16607535 ] Apache Spark commented on SPARK-23879: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23762) UTF8StringBuilder uses MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607538#comment-16607538 ] Apache Spark commented on SPARK-23762: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

[ https://issues.apache.org/jira/browse/SPARK-23879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607534#comment-16607534 ] Apache Spark commented on SPARK-23879: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607533#comment-16607533 ] Apache Spark commented on SPARK-10399: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607532#comment-16607532 ] Apache Spark commented on SPARK-10399: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607473#comment-16607473 ] Imran Rashid commented on SPARK-25344: -- > I actually roughly tried this and then quit before since

[jira] [Updated] (SPARK-25344) Break large tests.py files into smaller files

[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25344: - Labels: (was: newbie) > Break large tests.py files into smaller files >

[jira] [Updated] (SPARK-25371) Vector Assembler with no input columns leads to opaque error

[ https://issues.apache.org/jira/browse/SPARK-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Alor updated SPARK-25371: Summary: Vector Assembler with no input columns leads to opaque error (was: Vector Assembler

[jira] [Resolved] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah resolved SPARK-23429. Resolution: Fixed > Add executor memory metrics to heartbeat and expose in executors REST API >

[jira] [Updated] (SPARK-25371) Vector Assembler with no input columns throws an exception

[ https://issues.apache.org/jira/browse/SPARK-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Alor updated SPARK-25371: Description: When `VectorAssembler ` is given an empty array as its inputColumns it throws an

[jira] [Created] (SPARK-25371) Vector Assembler with no input columns throws an exception

Victor Alor created SPARK-25371: --- Summary: Vector Assembler with no input columns throws an exception Key: SPARK-25371 URL: https://issues.apache.org/jira/browse/SPARK-25371 Project: Spark

[jira] [Assigned] (SPARK-25015) Update Hadoop 2.7 to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25015: - Assignee: (was: Sean Owen) Target Version/s: (was: 2.4.0) Fix

[jira] [Reopened] (SPARK-25015) Update Hadoop 2.7 to 2.7.7

[ https://issues.apache.org/jira/browse/SPARK-25015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-25015: --- This was reverted in SPARK-25330 > Update Hadoop 2.7 to 2.7.7 > -- > >

[jira] [Updated] (SPARK-25309) Sci-kit Learn like Auto Pipeline Parallelization in Spark

[ https://issues.apache.org/jira/browse/SPARK-25309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-25309: - Component/s: ML > Sci-kit Learn like Auto Pipeline Parallelization in Spark >

[jira] [Assigned] (SPARK-25270) lint-python: Add flake8 to find syntax errors and undefined names

[ https://issues.apache.org/jira/browse/SPARK-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-25270: --- Assignee: cclauss > lint-python: Add flake8 to find syntax errors and undefined names >

[jira] [Resolved] (SPARK-25370) Undefined name _exception_message in java_gateway

[ https://issues.apache.org/jira/browse/SPARK-25370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-25370. - Resolution: Duplicate Issue was already fixed later. > Undefined name _exception_message in

[jira] [Created] (SPARK-25370) Undefined name _exception_message in java_gateway

holdenk created SPARK-25370: --- Summary: Undefined name _exception_message in java_gateway Key: SPARK-25370 URL: https://issues.apache.org/jira/browse/SPARK-25370 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-25370) Undefined name _exception_message in java_gateway

[ https://issues.apache.org/jira/browse/SPARK-25370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-25370: --- Assignee: holdenk > Undefined name _exception_message in java_gateway >

[jira] [Commented] (SPARK-1239) Improve fetching of map output statuses

[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607255#comment-16607255 ] harel gliksman commented on SPARK-1239: --- I am hitting a similar error on Spark 2.3.1 running on EMR

[jira] [Updated] (SPARK-25361) Support for Kinesis Client Library 2.0

[ https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cory Locklear updated SPARK-25361: -- Description: Amazon has recently released version 2.0 of the KCL which provides a HTTP/2

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Fix Version/s: 2.3.2 > Stage page aggregated executor metrics wrong when failures >

[jira] [Created] (SPARK-25369) Replace Java shim functional interfaces like java.api.Function with Java 8 equivalents

Sean Owen created SPARK-25369: - Summary: Replace Java shim functional interfaces like java.api.Function with Java 8 equivalents Key: SPARK-25369 URL: https://issues.apache.org/jira/browse/SPARK-25369

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Description: there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5) the

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns wrong result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Summary: Incorrect predicate pushdown returns wrong result (was: Incorrect predicate pushdown

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Summary: Incorrect predicate pushdown returns incorrect result (was: Incorrect predicate pushdown

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Description: there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5) the

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Attachment: (was: plan) > Incorrect predicate pushdown returns in incorrect result >

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Attachment: plan.txt > Incorrect predicate pushdown returns in incorrect result >

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Attachment: plan > Incorrect predicate pushdown returns in incorrect result >

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Description: there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5) the

[jira] [Created] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

Lev Katzav created SPARK-25368: -- Summary: Incorrect predicate pushdown returns in incorrect result Key: SPARK-25368 URL: https://issues.apache.org/jira/browse/SPARK-25368 Project: Spark Issue

[jira] [Updated] (SPARK-25368) Incorrect predicate pushdown returns in incorrect result

[ https://issues.apache.org/jira/browse/SPARK-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Katzav updated SPARK-25368: --- Description: there is a breaking change in spark 2.3 (I checked on 2.3.1 and 2.3.2-rc5) the

[jira] [Commented] (SPARK-25364) a better way to handle vector index and sparsity in FeatureHasher implementation ?

[ https://issues.apache.org/jira/browse/SPARK-25364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607019#comment-16607019 ] Marco Gaido commented on SPARK-25364: - Seems you created 2 JIRAs which are the same, if that is the

[jira] [Updated] (SPARK-25367) Hive table created by Spark dataFrame has incompatiable schema in spark and hive

[ https://issues.apache.org/jira/browse/SPARK-25367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yy updated SPARK-25367: --- Description: We save the dataframe object as a hive table in orc/parquet format in the spark shell. After we

[jira] [Updated] (SPARK-25367) Hive table created by Spark dataFrame has incompatiable schema in spark and hive

[ https://issues.apache.org/jira/browse/SPARK-25367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yy updated SPARK-25367: --- Environment: spark2.2.1 hive1.2.1 was: spark2.2.1 hive1.2.1 spark conf: hive-site.xml configured the

[jira] [Created] (SPARK-25367) Hive table created by Spark dataFrame has incompatiable schema in spark and hive

yy created SPARK-25367: -- Summary: Hive table created by Spark dataFrame has incompatiable schema in spark and hive Key: SPARK-25367 URL: https://issues.apache.org/jira/browse/SPARK-25367 Project: Spark

[jira] [Commented] (SPARK-25313) Fix regression in FileFormatWriter output schema

[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606796#comment-16606796 ] Apache Spark commented on SPARK-25313: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606797#comment-16606797 ] Rob Vesse commented on SPARK-25262: --- [~mcheah] I would like to keep this open so we can have the

[jira] [Assigned] (SPARK-25366) Zstd and brotil CompressionCodec are not supported for parquet files

[ https://issues.apache.org/jira/browse/SPARK-25366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25366: Assignee: (was: Apache Spark) > Zstd and brotil CompressionCodec are not supported

[jira] [Assigned] (SPARK-25366) Zstd and brotil CompressionCodec are not supported for parquet files

[ https://issues.apache.org/jira/browse/SPARK-25366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25366: Assignee: Apache Spark > Zstd and brotil CompressionCodec are not supported for parquet

[jira] [Commented] (SPARK-25366) Zstd and brotil CompressionCodec are not supported for parquet files

[ https://issues.apache.org/jira/browse/SPARK-25366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606793#comment-16606793 ] Apache Spark commented on SPARK-25366: -- User '10110346' has created a pull request for this issue:

[jira] [Created] (SPARK-25366) Zstd and brotil CompressionCodec are not supported for parquet files

liuxian created SPARK-25366: --- Summary: Zstd and brotil CompressionCodec are not supported for parquet files Key: SPARK-25366 URL: https://issues.apache.org/jira/browse/SPARK-25366 Project: Spark

[jira] [Commented] (SPARK-23098) Migrate Kafka batch source to v2

[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606758#comment-16606758 ] Jose Torres commented on SPARK-23098: - SPARK-23362 migrated the (microbatch) streaming source. This

[jira] [Resolved] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

[ https://issues.apache.org/jira/browse/SPARK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25267. - Resolution: Fixed Assignee: Dilip Biswal Fix Version/s: 2.4.0 > Disable

[jira] [Commented] (SPARK-25365) a better way to handle vector index and sparsity in FeatureHasher implementation ?

[ https://issues.apache.org/jira/browse/SPARK-25365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606746#comment-16606746 ] Vincent commented on SPARK-25365: - [~nick.pentre...@gmail.com] Thanks. > a better way to handle vector

[jira] [Created] (SPARK-25364) a better way to handle vector index and sparsity in FeatureHasher implementation ?

Vincent created SPARK-25364: --- Summary: a better way to handle vector index and sparsity in FeatureHasher implementation ? Key: SPARK-25364 URL: https://issues.apache.org/jira/browse/SPARK-25364 Project:

[jira] [Created] (SPARK-25365) a better way to handle vector index and sparsity in FeatureHasher implementation ?

Vincent created SPARK-25365: --- Summary: a better way to handle vector index and sparsity in FeatureHasher implementation ? Key: SPARK-25365 URL: https://issues.apache.org/jira/browse/SPARK-25365 Project: