[jira] [Assigned] (SPARK-27493) Upgrade ASM to 7.1

2019-04-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27493: Assignee: Dongjoon Hyun > Upgrade ASM to 7.1 > -- > >

[jira] [Resolved] (SPARK-27493) Upgrade ASM to 7.1

2019-04-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27493. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24395

[jira] [Updated] (SPARK-27498) Built-in parquet code path does not respect hive.enforce.bucketing

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27498: -- Description: _Caveat: I can see how this could be intentional if Spark believes that the

[jira] [Updated] (SPARK-27498) Built-in parquet code path does not respect hive.enforce.bucketing

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27498: -- Description: _Caveat: I can see how this could be intentional if Spark believes that the

[jira] [Updated] (SPARK-27498) Built-in parquet code path does not respect hive.enforce.bucketing

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27498: -- Description: _Caveat: I can see how this could be intentional if Spark believes that the

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Commented] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2019-04-17 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820707#comment-16820707 ] Mike Chan commented on SPARK-25422: --- Will this problem potentially hitting Spark 2.3.1 as well? I have

[jira] [Updated] (SPARK-27441) Add read/write tests to Hive serde tables(include Parquet vectorized reader)

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27441: Summary: Add read/write tests to Hive serde tables(include Parquet vectorized reader) (was: Add

[jira] [Updated] (SPARK-27441) Add read/write tests to Hive serde tables

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27441: Issue Type: Sub-task (was: Improvement) Parent: SPARK-27500 > Add read/write tests to

[jira] [Created] (SPARK-27501) Add test for HIVE-13083: Writing HiveDecimal to ORC can wrongly suppress present stream

2019-04-17 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27501: --- Summary: Add test for HIVE-13083: Writing HiveDecimal to ORC can wrongly suppress present stream Key: SPARK-27501 URL: https://issues.apache.org/jira/browse/SPARK-27501

[jira] [Created] (SPARK-27500) Add tests for the built-in Hive 2.3

2019-04-17 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27500: --- Summary: Add tests for the built-in Hive 2.3 Key: SPARK-27500 URL: https://issues.apache.org/jira/browse/SPARK-27500 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

2019-04-17 Thread Junjie Chen (JIRA)
Junjie Chen created SPARK-27499: --- Summary: Support mapping spark.local.dir to hostPath volume Key: SPARK-27499 URL: https://issues.apache.org/jira/browse/SPARK-27499 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

2019-04-17 Thread william hesch (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820637#comment-16820637 ] william hesch commented on SPARK-17668: --- +1 > Support representing structs with case classes and

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820619#comment-16820619 ] Stavros Kontopoulos commented on SPARK-27491: - Can you reach the rest api outside spark

[jira] [Created] (SPARK-27498) Built-in parquet code path does not respect hive.enforce.bucketing

2019-04-17 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-27498: - Summary: Built-in parquet code path does not respect hive.enforce.bucketing Key: SPARK-27498 URL: https://issues.apache.org/jira/browse/SPARK-27498 Project: Spark

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Created] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-17 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-27497: - Summary: Spark wipes out bucket spec in metastore when updating table stats Key: SPARK-27497 URL: https://issues.apache.org/jira/browse/SPARK-27497 Project: Spark

[jira] [Comment Edited] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features

2019-04-17 Thread Calvin Park (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820549#comment-16820549 ] Calvin Park edited comment on SPARK-27365 at 4/17/19 10:34 PM: --- Simple

[jira] [Comment Edited] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features

2019-04-17 Thread Calvin Park (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820549#comment-16820549 ] Calvin Park edited comment on SPARK-27365 at 4/17/19 10:34 PM: --- Simple

[jira] [Commented] (SPARK-27365) Spark Jenkins supports testing GPU-aware scheduling features

2019-04-17 Thread Calvin Park (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820549#comment-16820549 ] Calvin Park commented on SPARK-27365: - Simple Jenkinsfile {code:java} pipeline { agent {

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820542#comment-16820542 ] t oo commented on SPARK-27491: -- cc: [~skonto] [~mpmolek] [~gschiavon] [~scrapco...@gmail.com] > SPARK REST

[jira] [Commented] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-17 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820525#comment-16820525 ] shahid commented on SPARK-27468: [~zsxwing] Thanks > "Storage Level" in "RDD Storage Page" is not

[jira] [Created] (SPARK-27496) RPC should send back the fatal errors

2019-04-17 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27496: Summary: RPC should send back the fatal errors Key: SPARK-27496 URL: https://issues.apache.org/jira/browse/SPARK-27496 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27434) memory leak in spark driver

2019-04-17 Thread Ryne Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820516#comment-16820516 ] Ryne Yang commented on SPARK-27434: --- [~shahid]   were you able to reproduce this by the steps I

[jira] [Commented] (SPARK-27473) Support filter push down for status fields in binary file data source

2019-04-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820500#comment-16820500 ] Xiangrui Meng commented on SPARK-27473: --- Given SPARK-25558 is WIP, we might want to flatten the

[jira] [Assigned] (SPARK-27473) Support filter push down for status fields in binary file data source

2019-04-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27473: - Assignee: Weichen Xu > Support filter push down for status fields in binary file data

[jira] [Created] (SPARK-27495) Support Stage level resource scheduling

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27495: - Summary: Support Stage level resource scheduling Key: SPARK-27495 URL: https://issues.apache.org/jira/browse/SPARK-27495 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: Support Stage level resource configuration and scheduling (was: Support Stage level

[jira] [Commented] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820460#comment-16820460 ] Shixiong Zhu commented on SPARK-27468: -- [~shahid] You need to use "--master

[jira] [Created] (SPARK-27494) Null values don't work in Kafka source v2

2019-04-17 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27494: Summary: Null values don't work in Kafka source v2 Key: SPARK-27494 URL: https://issues.apache.org/jira/browse/SPARK-27494 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27493) Upgrade ASM to 7.1

2019-04-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27493: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-24417 > Upgrade ASM to 7.1 >

[jira] [Created] (SPARK-27493) Upgrade ASM to 7.1

2019-04-17 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-27493: - Summary: Upgrade ASM to 7.1 Key: SPARK-27493 URL: https://issues.apache.org/jira/browse/SPARK-27493 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27276) Increase the minimum pyarrow version to 0.12.1

2019-04-17 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820370#comment-16820370 ] shane knapp commented on SPARK-27276: - test currently running:

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

2019-04-17 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820364#comment-16820364 ] shane knapp commented on SPARK-25079: - from my email to dev@: ok. after much wailing and gnashing

[jira] [Closed] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"

2019-04-17 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp closed SPARK-27389. --- > pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'" >

[jira] [Commented] (SPARK-25088) Rest Server default & doc updates

2019-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820321#comment-16820321 ] Imran Rashid commented on SPARK-25088: -- if you're allowing unauthed rest, what is the point of auth

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820309#comment-16820309 ] Imran Rashid commented on SPARK-6235: - [~glal14] actually this was fixed in 2.4. There was one open

[jira] [Resolved] (SPARK-6235) Address various 2G limits

2019-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6235. - Resolution: Fixed Fix Version/s: 2.4.0 > Address various 2G limits >

[jira] [Resolved] (SPARK-24936) Better error message when trying a shuffle fetch over 2 GB

2019-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-24936. -- Resolution: Won't Fix As we've already shipped 2.4, I think its unlikely we're going to fix

[jira] [Created] (SPARK-27492) High level user documentation

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27492: - Summary: High level user documentation Key: SPARK-27492 URL: https://issues.apache.org/jira/browse/SPARK-27492 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27403) Fix `updateTableStats` to update table stats always with new stats or None

2019-04-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27403: -- Summary: Fix `updateTableStats` to update table stats always with new stats or None (was:

[jira] [Updated] (SPARK-27403) Failed to update the table size automatically even though spark.sql.statistics.size.autoUpdate.enabled is set as rue

2019-04-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27403: -- Fix Version/s: 2.4.2 > Failed to update the table size automatically even though >

[jira] [Commented] (SPARK-6235) Address various 2G limits

2019-04-17 Thread Gowtam Lal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820229#comment-16820229 ] Gowtam Lal commented on SPARK-6235: --- It would be great to see this go out. Any updates? > Address

[jira] [Updated] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27491: - Summary: SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820211#comment-16820211 ] t oo commented on SPARK-27491: -- cc: [~ash] [~bolke] > SPARK REST API -

[jira] [Created] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark

2019-04-17 Thread t oo (JIRA)
t oo created SPARK-27491: Summary: SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark Key: SPARK-27491 URL:

[jira] [Commented] (SPARK-27485) Certain query plans fail to run when autoBroadcastJoinThreshold is set to -1

2019-04-17 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820189#comment-16820189 ] shahid commented on SPARK-27485: Could you please share test to reproduce this? > Certain query plans

[jira] [Created] (SPARK-27490) File source V2: return correct result for Dataset.inputFiles()

2019-04-17 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-27490: -- Summary: File source V2: return correct result for Dataset.inputFiles() Key: SPARK-27490 URL: https://issues.apache.org/jira/browse/SPARK-27490 Project: Spark

[jira] [Created] (SPARK-27489) UI updates to show executor resource information

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27489: - Summary: UI updates to show executor resource information Key: SPARK-27489 URL: https://issues.apache.org/jira/browse/SPARK-27489 Project: Spark Issue

[jira] [Reopened] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-27364: --- reopening since it has a subtask > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820120#comment-16820120 ] Thomas Graves commented on SPARK-27364: --- based on no comments on this I'm going to resolve this

[jira] [Resolved] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27364. --- Resolution: Fixed > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820119#comment-16820119 ] Thomas Graves commented on SPARK-27488: --- Note, the api design is here:

[jira] [Created] (SPARK-27488) Driver interface to support GPU resources

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27488: - Summary: Driver interface to support GPU resources Key: SPARK-27488 URL: https://issues.apache.org/jira/browse/SPARK-27488 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2019-04-17 Thread Dave DeCaprio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820114#comment-16820114 ] Dave DeCaprio commented on SPARK-23904: --- No, it's just in master, which is the 3.X branch. I do

[jira] [Resolved] (SPARK-27458) Remind developer using IntelliJ to update maven version

2019-04-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27458. --- Resolution: Fixed Assignee: William Wong Fix Version/s: 3.0.0 Resolved by

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2019-04-17 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820058#comment-16820058 ] Izek Greenfield commented on SPARK-23904: - [~DaveDeCaprio] Does that PR go into 2.4.1 release?

[jira] [Resolved] (SPARK-19712) EXISTS and Left Semi join do not produce the same plan

2019-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19712. - Resolution: Fixed Issue resolved by pull request 24331

[jira] [Assigned] (SPARK-19712) EXISTS and Left Semi join do not produce the same plan

2019-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19712: --- Assignee: Dilip Biswal > EXISTS and Left Semi join do not produce the same plan >

[jira] [Issue Comment Deleted] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2019-04-17 Thread Zhanfeng Huo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanfeng Huo updated SPARK-3438: Comment: was deleted (was: This is a newest PR on master with commit

[jira] [Issue Comment Deleted] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2019-04-17 Thread Zhanfeng Huo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanfeng Huo updated SPARK-3438: Comment: was deleted (was: test) > Support for accessing secured HDFS in Standalone Mode >

[jira] [Commented] (SPARK-3438) Support for accessing secured HDFS in Standalone Mode

2019-04-17 Thread Zhanfeng Huo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820005#comment-16820005 ] Zhanfeng Huo commented on SPARK-3438: - test > Support for accessing secured HDFS in Standalone Mode

[jira] [Resolved] (SPARK-27430) broadcast hint should be respected for broadcast nested loop join

2019-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27430. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24376

[jira] [Created] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-17 Thread Vadym Holubnychyi (JIRA)
Vadym Holubnychyi created SPARK-27487: - Summary: Spark - Scala 2.12 compatibility Key: SPARK-27487 URL: https://issues.apache.org/jira/browse/SPARK-27487 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19842) Informational Referential Integrity Constraints Support in Spark

2019-04-17 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819844#comment-16819844 ] Song Jun commented on SPARK-19842: -- I think Constraint should be designed with DataSource v2 and can do

[jira] [Commented] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-17 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819809#comment-16819809 ] Gabor Somogyi commented on SPARK-27409: --- I mean does this cause any data processing issue other

[jira] [Updated] (SPARK-27475) dev/deps/spark-deps-hadoop-3.2 is incorrect

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27475: Issue Type: Sub-task (was: Bug) Parent: SPARK-23710 > dev/deps/spark-deps-hadoop-3.2 is

[jira] [Updated] (SPARK-27402) Fix hadoop-3.2 test issue(except the hive-thriftserver module)

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27402: Description: Fix sql/core and sql/hive modules test issue for hadoop-3.2 > Fix hadoop-3.2 test

[jira] [Updated] (SPARK-27402) Fix hadoop-3.2 test issue(except the hive-thriftserver module)

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27402: Description: (was: When we upgrade the built-in Hive to 2.3.4, the default

[jira] [Updated] (SPARK-27402) Fix hadoop-3.2 test issue(except the hive-thriftserver module)

2019-04-17 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27402: Summary: Fix hadoop-3.2 test issue(except the hive-thriftserver module) (was: Support

[jira] [Commented] (SPARK-25088) Rest Server default & doc updates

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819764#comment-16819764 ] t oo commented on SPARK-25088: -- why block rest if auth is on? for example i want to be able to use unauthed

[jira] [Created] (SPARK-27486) Enable History server storage information test

2019-04-17 Thread shahid (JIRA)
shahid created SPARK-27486: -- Summary: Enable History server storage information test Key: SPARK-27486 URL: https://issues.apache.org/jira/browse/SPARK-27486 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27486) Enable History server storage information test

2019-04-17 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819759#comment-16819759 ] shahid commented on SPARK-27486: I will raise a PR > Enable History server storage information test >