[jira] [Commented] (SPARK-28333) NULLS FIRST for DESC and NULLS LAST for ASC

2019-07-12 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884292#comment-16884292 ] Shivu Sondur commented on SPARK-28333: -- [~yumwang] i am working on this > NULLS FIRST for DESC

[jira] [Commented] (SPARK-25411) Implement range partition in Spark

2019-07-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884290#comment-16884290 ] Yuming Wang commented on SPARK-25411: - PostgreSQL support range Partition.

[jira] [Updated] (SPARK-25411) Implement range partition in Spark

2019-07-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25411: Issue Type: Sub-task (was: New Feature) Parent: SPARK-27764 > Implement range partition

[jira] [Created] (SPARK-28377) Fully support correlation names in the FROM clause

2019-07-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28377: --- Summary: Fully support correlation names in the FROM clause Key: SPARK-28377 URL: https://issues.apache.org/jira/browse/SPARK-28377 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-28288) Convert and port 'window.sql' into UDF test base

2019-07-12 Thread YoungGyu Chun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884259#comment-16884259 ] YoungGyu Chun edited comment on SPARK-28288 at 7/13/19 1:21 AM: Hello

[jira] [Commented] (SPARK-28288) Convert and port 'window.sql' into UDF test base

2019-07-12 Thread YoungGyu Chun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884259#comment-16884259 ] YoungGyu Chun commented on SPARK-28288: --- Hello [~hyukjin.kwon], The following is the results of

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/13/19 12:37 AM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/13/19 12:37 AM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/13/19 12:36 AM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/13/19 12:35 AM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/13/19 12:34 AM:

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884250#comment-16884250 ] Stavros Kontopoulos commented on SPARK-27927: - It is on 2.4.0:

[jira] [Resolved] (SPARK-28361) Test equality of generated code with id in class name

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28361. --- Resolution: Fixed Fix Version/s: 2.4.4 2.3.4

[jira] [Created] (SPARK-28376) Write sorted parquet files

2019-07-12 Thread t oo (JIRA)
t oo created SPARK-28376: Summary: Write sorted parquet files Key: SPARK-28376 URL: https://issues.apache.org/jira/browse/SPARK-28376 Project: Spark Issue Type: New Feature Components:

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Edwin Biemond (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884223#comment-16884223 ] Edwin Biemond commented on SPARK-27927: --- We also have the exact same issue/ behaviour  on spark

[jira] [Commented] (SPARK-15420) Repartition and sort before Parquet writes

2019-07-12 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884220#comment-16884220 ] t oo commented on SPARK-15420: -- PR was abandoned :( > Repartition and sort before Parquet writes >

[jira] [Updated] (SPARK-28375) Enforce idempotence on the PullupCorrelatedPredicates optimizer rule

2019-07-12 Thread Yesheng Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesheng Ma updated SPARK-28375: --- Summary: Enforce idempotence on the PullupCorrelatedPredicates optimizer rule (was: Fix

[jira] [Created] (SPARK-28375) Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence

2019-07-12 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28375: -- Summary: Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence Key: SPARK-28375 URL: https://issues.apache.org/jira/browse/SPARK-28375 Project: Spark

[jira] [Created] (SPARK-28374) DataSourceV2: Add method to support INSERT ... IF NOT EXISTS

2019-07-12 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-28374: - Summary: DataSourceV2: Add method to support INSERT ... IF NOT EXISTS Key: SPARK-28374 URL: https://issues.apache.org/jira/browse/SPARK-28374 Project: Spark

[jira] [Commented] (SPARK-24663) Flaky test: StreamingContextSuite "stop slow receiver gracefully"

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884196#comment-16884196 ] Dongjoon Hyun commented on SPARK-24663: --- I hit the same issue in Riselab Jenkins today. -

[jira] [Updated] (SPARK-28361) Test equality of generated code with id in class name

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28361: -- Affects Version/s: 2.3.3 2.4.3 > Test equality of generated code with

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884121#comment-16884121 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/12/19 7:41 PM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884121#comment-16884121 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/12/19 7:40 PM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884121#comment-16884121 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/12/19 7:40 PM:

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884121#comment-16884121 ] Stavros Kontopoulos commented on SPARK-27927: - I think the issue is here: ```

[jira] [Commented] (SPARK-25080) NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110)

2019-07-12 Thread Serge Shikov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884113#comment-16884113 ] Serge Shikov commented on SPARK-25080: -- Sorry, I have no other versions available at the moment.

[jira] [Updated] (SPARK-28372) Document Spark WEB UI

2019-07-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-28372: Target Version/s: 3.0.0 > Document Spark WEB UI > - > > Key:

[jira] [Created] (SPARK-28373) Document JDBC/ODBC Server page

2019-07-12 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28373: --- Summary: Document JDBC/ODBC Server page Key: SPARK-28373 URL: https://issues.apache.org/jira/browse/SPARK-28373 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-28372) Document Spark WEB UI

2019-07-12 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28372: --- Summary: Document Spark WEB UI Key: SPARK-28372 URL: https://issues.apache.org/jira/browse/SPARK-28372 Project: Spark Issue Type: Umbrella Components:

[jira] [Assigned] (SPARK-28371) Parquet "starts with" filter is not null-safe

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28371: Assignee: (was: Apache Spark) > Parquet "starts with" filter is not null-safe >

[jira] [Assigned] (SPARK-28371) Parquet "starts with" filter is not null-safe

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28371: Assignee: Apache Spark > Parquet "starts with" filter is not null-safe >

[jira] [Resolved] (SPARK-28260) Add CLOSED state to ExecutionState

2019-07-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28260. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 3.0.0 > Add CLOSED state to

[jira] [Updated] (SPARK-28371) Parquet "starts with" filter is not null-safe

2019-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-28371: --- Description: I ran into this when running unit tests with Parquet 1.11. It seems that 1.10

[jira] [Created] (SPARK-28371) Parquet "starts with" filter is not null-safe

2019-07-12 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-28371: -- Summary: Parquet "starts with" filter is not null-safe Key: SPARK-28371 URL: https://issues.apache.org/jira/browse/SPARK-28371 Project: Spark Issue

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: The some python tests fail with `error: release unlocked lock`. -

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Priority: Blocker (was: Major) > Fix Flaky Tests - test_time_with_timezone >

[jira] [Commented] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883993#comment-16883993 ] Dongjoon Hyun commented on SPARK-28358: --- Hi, [~hyukjin.kwon]. Could you take a look at this

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: -

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: -

[jira] [Updated] (SPARK-28266) data duplication when `path` serde property is present

2019-07-12 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-28266: -- Summary: data duplication when `path` serde property is present (was: data

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Edwin Biemond (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883974#comment-16883974 ] Edwin Biemond commented on SPARK-27927: --- I added them as an attachement to this ticket. see above

[jira] [Assigned] (SPARK-28370) Upgrade Mockito to 2.28.2

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28370: Assignee: Apache Spark > Upgrade Mockito to 2.28.2 > - > >

[jira] [Assigned] (SPARK-28370) Upgrade Mockito to 2.28.2

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28370: Assignee: (was: Apache Spark) > Upgrade Mockito to 2.28.2 >

[jira] [Created] (SPARK-28370) Upgrade Mockito to 2.28.2

2019-07-12 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-28370: - Summary: Upgrade Mockito to 2.28.2 Key: SPARK-28370 URL: https://issues.apache.org/jira/browse/SPARK-28370 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-28199) Move Trigger implementations to Triggers.scala and avoid exposing these to the end users

2019-07-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28199: - Docs Text: In Spark 3.0, the deprecated class org.apache.spark.sql.streaming.ProcessingTime

[jira] [Assigned] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26175: Assignee: (was: Apache Spark) > PySpark cannot terminate worker process if user

[jira] [Assigned] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26175: Assignee: Apache Spark > PySpark cannot terminate worker process if user program reads

[jira] [Assigned] (SPARK-28348) Avoid cast twice for decimal type

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28348: Assignee: (was: Apache Spark) > Avoid cast twice for decimal type >

[jira] [Assigned] (SPARK-28348) Avoid cast twice for decimal type

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28348: Assignee: Apache Spark > Avoid cast twice for decimal type >

[jira] [Assigned] (SPARK-28322) DIV support decimal type

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28322: Assignee: (was: Apache Spark) > DIV support decimal type >

[jira] [Assigned] (SPARK-28322) DIV support decimal type

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28322: Assignee: Apache Spark > DIV support decimal type > > >

[jira] [Resolved] (SPARK-28228) Fix substitution order of nested WITH clauses

2019-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28228. --- Resolution: Fixed Assignee: Peter Toth Fix Version/s: 3.0.0 This is

[jira] [Updated] (SPARK-28224) Sum aggregation returns null on overflow decimals

2019-07-12 Thread Mick Jermsurawong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mick Jermsurawong updated SPARK-28224: -- Description: To reproduce: {code:java} import spark.implicits._ val ds = spark  

[jira] [Updated] (SPARK-28224) Check overflow in decimal Sum aggregate

2019-07-12 Thread Mick Jermsurawong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mick Jermsurawong updated SPARK-28224: -- Summary: Check overflow in decimal Sum aggregate (was: Sum aggregation returns null

[jira] [Created] (SPARK-28369) Check overflow in decimal UDF

2019-07-12 Thread Mick Jermsurawong (JIRA)
Mick Jermsurawong created SPARK-28369: - Summary: Check overflow in decimal UDF Key: SPARK-28369 URL: https://issues.apache.org/jira/browse/SPARK-28369 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28369) Check overflow in decimal UDF

2019-07-12 Thread Mick Jermsurawong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mick Jermsurawong updated SPARK-28369: -- Description: Udf resulting in overflowing BigDecimal currently returns null. This is

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883814#comment-16883814 ] Stavros Kontopoulos commented on SPARK-27927: - [~ebiemond] could you paste the rest of the

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883814#comment-16883814 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/12/19 1:36 PM:

[jira] [Comment Edited] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883814#comment-16883814 ] Stavros Kontopoulos edited comment on SPARK-27927 at 7/12/19 1:36 PM:

[jira] [Updated] (SPARK-28368) Row.getAs() return different values in scala and java

2019-07-12 Thread Kideok Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kideok Kim updated SPARK-28368: --- Description: *description* * About

[jira] [Updated] (SPARK-28368) Row.getAs() return different values in scala and java

2019-07-12 Thread Kideok Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kideok Kim updated SPARK-28368: --- Description: *description* * About

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Edwin Biemond (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883801#comment-16883801 ] Edwin Biemond commented on SPARK-27927: --- I added those threads dumps and I see this    {noformat}

[jira] [Updated] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Edwin Biemond (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwin Biemond updated SPARK-27927: -- Attachment: driver_threads.log executor_threads.log > driver pod hangs with

[jira] [Updated] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2019-07-12 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28367: -- Affects Version/s: 2.1.3 2.2.3 2.3.3

[jira] [Created] (SPARK-28368) Row.getAs() return different values in scala and java

2019-07-12 Thread Kideok Kim (JIRA)
Kideok Kim created SPARK-28368: -- Summary: Row.getAs() return different values in scala and java Key: SPARK-28368 URL: https://issues.apache.org/jira/browse/SPARK-28368 Project: Spark Issue

[jira] [Assigned] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28367: Assignee: Apache Spark > Kafka connector infinite wait because metadata never updated >

[jira] [Assigned] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28367: Assignee: (was: Apache Spark) > Kafka connector infinite wait because metadata never

[jira] [Created] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2019-07-12 Thread Gabor Somogyi (JIRA)
Gabor Somogyi created SPARK-28367: - Summary: Kafka connector infinite wait because metadata never updated Key: SPARK-28367 URL: https://issues.apache.org/jira/browse/SPARK-28367 Project: Spark

[jira] [Assigned] (SPARK-28366) Logging in driver when loading single large gzipped file via sc.textFile

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28366: Assignee: (was: Apache Spark) > Logging in driver when loading single large gzipped

[jira] [Assigned] (SPARK-28366) Logging in driver when loading single large gzipped file via sc.textFile

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28366: Assignee: Apache Spark > Logging in driver when loading single large gzipped file via

[jira] [Created] (SPARK-28366) Logging in driver when loading single large gzipped file via sc.textFile

2019-07-12 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-28366: -- Summary: Logging in driver when loading single large gzipped file via sc.textFile Key: SPARK-28366 URL: https://issues.apache.org/jira/browse/SPARK-28366 Project: Spark

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Edwin Biemond (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883732#comment-16883732 ] Edwin Biemond commented on SPARK-27927: --- today I verified this again with the latest of master and

[jira] [Commented] (SPARK-27927) driver pod hangs with pyspark 2.4.3 and master on kubenetes

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883729#comment-16883729 ] Stavros Kontopoulos commented on SPARK-27927: - Could this be related to this: 

[jira] [Comment Edited] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883725#comment-16883725 ] Stavros Kontopoulos edited comment on SPARK-26833 at 7/12/19 11:32 AM:

[jira] [Commented] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883725#comment-16883725 ] Stavros Kontopoulos commented on SPARK-26833: - I think the sa account is meant for anything

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:24 AM:

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:20 AM:

[jira] [Assigned] (SPARK-28365) Set default locale for StopWordsRemover tests to prevent invalid locale error during test

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28365: Assignee: Apache Spark > Set default locale for StopWordsRemover tests to prevent

[jira] [Assigned] (SPARK-28365) Set default locale for StopWordsRemover tests to prevent invalid locale error during test

2019-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28365: Assignee: (was: Apache Spark) > Set default locale for StopWordsRemover tests to

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:19 AM:

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:18 AM:

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:16 AM:

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos edited comment on SPARK-27812 at 7/12/19 11:12 AM:

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883716#comment-16883716 ] Stavros Kontopoulos commented on SPARK-27812: - There is a bigger issue in general with the 

[jira] [Created] (SPARK-28365) Set default locale for StopWordsRemover tests to prevent invalid locale error during test

2019-07-12 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-28365: --- Summary: Set default locale for StopWordsRemover tests to prevent invalid locale error during test Key: SPARK-28365 URL: https://issues.apache.org/jira/browse/SPARK-28365

[jira] [Comment Edited] (SPARK-27997) kubernetes client token expired

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos edited comment on SPARK-27997 at 7/12/19 11:05 AM:

[jira] [Comment Edited] (SPARK-27997) kubernetes client token expired

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos edited comment on SPARK-27997 at 7/12/19 11:04 AM:

[jira] [Comment Edited] (SPARK-27997) kubernetes client token expired

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos edited comment on SPARK-27997 at 7/12/19 11:01 AM:

[jira] [Comment Edited] (SPARK-27997) kubernetes client token expired

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos edited comment on SPARK-27997 at 7/12/19 10:58 AM:

[jira] [Commented] (SPARK-27997) kubernetes client token expired

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos commented on SPARK-27997: - This is interesting. With K8s 1.16 sa tokens

[jira] [Commented] (SPARK-28279) Convert and port 'group-analytics.sql' into UDF test base

2019-07-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883692#comment-16883692 ] Stavros Kontopoulos commented on SPARK-28279: - Waiting for 

[jira] [Resolved] (SPARK-28354) Use JIRA user name instead of JIRA user key

2019-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28354. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25120

[jira] [Assigned] (SPARK-28357) Fix Flaky Test - FileAppenderSuite.rolling file appender - size-based rolling compressed

2019-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28357: Assignee: Dongjoon Hyun > Fix Flaky Test - FileAppenderSuite.rolling file appender -

[jira] [Resolved] (SPARK-28357) Fix Flaky Test - FileAppenderSuite.rolling file appender - size-based rolling compressed

2019-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28357. -- Resolution: Fixed Fix Version/s: 2.4.4 2.3.4

[jira] [Resolved] (SPARK-28321) functions.udf(UDF0, DataType) produces unexpected results

2019-07-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28321. - Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 3.0.0 >

[jira] [Updated] (SPARK-28364) Unable to read complete data from an external hive table stored as ORC that points to a managed table's data files which is getting stored in sub-directories.

2019-07-12 Thread Debdut Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debdut Mukherjee updated SPARK-28364: - Description: Unable to read complete data from an external hive table stored as ORC

[jira] [Updated] (SPARK-28364) Unable to read complete data from an external hive table stored as ORC that points to a managed table's data files which is getting stored in sub-directories.

2019-07-12 Thread Debdut Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debdut Mukherjee updated SPARK-28364: - Description: Unable to read complete data from an external hive table stored as ORC

[jira] [Updated] (SPARK-28364) Unable to read complete data from an external hive table stored as ORC that points to a managed table's data files which is getting stored in sub-directories.

2019-07-12 Thread Debdut Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debdut Mukherjee updated SPARK-28364: - Description: Unable to read complete data from an external hive table stored as ORC

[jira] [Updated] (SPARK-28364) Unable to read complete data from an external hive table stored as ORC that points to a managed table's data files which is getting stored in sub-directories.

2019-07-12 Thread Debdut Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debdut Mukherjee updated SPARK-28364: - Description: Unable to read complete data from an external hive table stored as ORC

[jira] [Updated] (SPARK-28364) Unable to read complete data from an external hive table stored as ORC that points to a managed table's data files which is getting stored in sub-directories.

2019-07-12 Thread Debdut Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Debdut Mukherjee updated SPARK-28364: - Description: Unable to read complete data from an external hive table stored as ORC

  1   2   >