[jira] [Updated] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26999: Attachment: SPARK-26999.png > SparkSQL CLIDriver parses sql statement incorrectly >

[jira] [Resolved] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-27 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang resolved SPARK-26999. - Resolution: Duplicate > SparkSQL CLIDriver parses sql statement incorrectly >

[jira] [Created] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-26 Thread feiwang (JIRA)
feiwang created SPARK-26999: --- Summary: SparkSQL CLIDriver parses sql statement incorrectly Key: SPARK-26999 URL: https://issues.apache.org/jira/browse/SPARK-26999 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26999: Description: SparkSQLCLIDriver parse sql statement incorrectly, because its processLine method is not

[jira] [Updated] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26999: Description: SparkSQLCLIDriver parse sql statement incorrectly, because its processLine method is not

[jira] [Updated] (SPARK-26999) SparkSQL CLIDriver parses sql statement incorrectly

2019-02-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26999: Description: SparkSQLCLIDriver parse sql statement incorrectly, because its processLine method is not

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2019-03-12 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790258#comment-16790258 ] feiwang commented on SPARK-4105: Is there anyone see these errors in latest version such as spark-2.3?

[jira] [Comment Edited] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2019-03-12 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790258#comment-16790258 ] feiwang edited comment on SPARK-4105 at 3/12/19 6:03 AM: - Is there anyone see

[jira] [Updated] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27247: Description: Catalyst cache some logic plan. During relationConversions(Relation conversion from

[jira] [Created] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
feiwang created SPARK-27247: --- Summary: [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly Key: SPARK-27247 URL: https://issues.apache.org/jira/browse/SPARK-27247

[jira] [Updated] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27247: Description: Catalyst cache some logic plan. During relationConversions(Relation conversion from

[jira] [Closed] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-02-11 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang closed SPARK-26795. --- > Retry remote fileSegmentManagedBuffer when creating inputStream failed during > shuffle read phase >

[jira] [Resolved] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-02-11 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang resolved SPARK-26795. - Resolution: Not A Problem > Retry remote fileSegmentManagedBuffer when creating inputStream failed

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Fix Version/s: (was: 2.4.1) 2.3.0 2.3.1 2.3.2

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Summary: Retry remote fileSegmentManagedBuffer when create inputStream failed during shuffle read phase

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Flags: Patch Target Version/s: 2.4.0, 2.3.2 (was: 2.3.2, 2.4.0) > Retry remote

[jira] [Updated] (SPARK-26795) During shuffle read phase, the managedBuffer which throw IOException may be a remote download FileSegment and should be retry instead of throwFetchFailed directly.

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Shepherd: (was: feiwang) Target Version/s: 2.4.0, 2.3.2 (was: 2.3.2, 2.4.0) > During

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Target Version/s: 2.4.0, 2.3.2 (was: 2.3.2, 2.4.0) Fix Version/s: (was: 2.3.2)

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Affects Version/s: 2.4.0 > Retry remote fileSegmentManagedBuffer when create inputStream during shuffle

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Summary: Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase (was:

[jira] [Created] (SPARK-26795) During shuffle read phase, the managedBuffer which throw IOException may be a remote download FileSegment and should be retry instead of throwFetchFailed directly.

2019-01-31 Thread feiwang (JIRA)
feiwang created SPARK-26795: --- Summary: During shuffle read phase, the managedBuffer which throw IOException may be a remote download FileSegment and should be retry instead of throwFetchFailed directly. Key: SPARK-26795

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when create inputStream during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Description: During shuffle read phase, theĀ  managedBuffer which throw IOException may be a remote

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Summary: Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Description: There is a parameter spark.maxRemoteBlockSizeFetchToMem, which means the remote block will

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Description: There is a parameter `spark.maxRemoteBlockSizeFetchToMem`, which means the remote block

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Description: There is a parameter *spark.maxRemoteBlockSizeFetchToMem*, which means the remote block

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Description: There is a parameter spark.maxRemoteBlockSizeFetchToMem, which means the remote block will

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Target Version/s: 2.4.0, 2.3.2 (was: 2.3.2, 2.4.0) Labels: shuffle (was: ) > Retry remote

[jira] [Updated] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-01-31 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-26795: Labels: (was: shuffle) > Retry remote fileSegmentManagedBuffer when creating inputStream failed during

[jira] [Updated] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-12 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27637: Affects Version/s: (was: 2.4.2) (was: 2.3.2) 2.3.3

[jira] [Updated] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-12 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27637: Component/s: (was: Block Manager) > If exception occured while fetching blocks by netty block

[jira] [Updated] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-12 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27637: Component/s: Shuffle > If exception occured while fetching blocks by netty block transfer service, >

[jira] [Commented] (SPARK-27614) Executor shuffle fetch hang

2019-05-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848388#comment-16848388 ] feiwang commented on SPARK-27614: - You can propose a PR to spark. It seems meaningful. > Executor

[jira] [Issue Comment Deleted] (SPARK-27614) Executor shuffle fetch hang

2019-05-26 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27614: Comment: was deleted (was: You can propose a PR to spark. It seems meaningful.) > Executor shuffle fetch

[jira] [Updated] (SPARK-27876) Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

2019-05-30 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27876: Affects Version/s: (was: 3.1.0) (was: 2.4.3) 2.3.2

[jira] [Updated] (SPARK-27876) [CORE][SHUFFLE] Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

2019-05-29 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27876: Description: There is a limit for shuffle read. If a shuffle partition block's size is large than

[jira] [Updated] (SPARK-27876) [CORE][SHUFFLE] Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

2019-05-29 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27876: Description: There is a limit for shuffle read. If a shuffle partition block's size is large than

[jira] [Created] (SPARK-27876) [CORE][SHUFFLE] Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

2019-05-29 Thread feiwang (JIRA)
feiwang created SPARK-27876: --- Summary: [CORE][SHUFFLE] Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block. Key: SPARK-27876 URL:

[jira] [Created] (SPARK-27716) Complete the transactions support for part of jdbc datasource operation

2019-05-15 Thread feiwang (JIRA)
feiwang created SPARK-27716: --- Summary: Complete the transactions support for part of jdbc datasource operation Key: SPARK-27716 URL: https://issues.apache.org/jira/browse/SPARK-27716 Project: Spark

[jira] [Updated] (SPARK-27716) Complete the transactions support for part of jdbc datasource operations.

2019-05-15 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27716: Summary: Complete the transactions support for part of jdbc datasource operations. (was: Complete the

[jira] [Updated] (SPARK-27716) Complete the transactions support for part of jdbc datasource operations.

2019-05-15 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27716: Description: With the jdbc datasource, we can save a rdd to the database. The comments for the function

[jira] [Updated] (SPARK-27716) Complete the transactions support for part of jdbc datasource operations.

2019-05-15 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27716: Description: With the jdbc datasource, we can save a rdd to the database. The comments for the function

[jira] [Updated] (SPARK-27716) Complete the transactions support for part of jdbc datasource operations.

2019-05-15 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27716: Labels: pull-request-available (was: ) > Complete the transactions support for part of jdbc datasource

[jira] [Updated] (SPARK-27814) The cast operation for partitioned column may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27814: Description: For a partitioned table, such as: {code:sql} table test (c1 int, c2 string) partitioned by

[jira] [Updated] (SPARK-27814) The cast operation for partitioned column may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27814: Description: For a partitioned table, such as: {code:sql} table test (c1 int, c2 string) partitioned by

[jira] [Updated] (SPARK-27814) The cast operation for partitioned column may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27814: Description: For a partitioned table, such as: table test (c1 int, c2 string) partitioned by (c3 Int) If

[jira] [Commented] (SPARK-27814) The cast operation may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846544#comment-16846544 ] feiwang commented on SPARK-27814: - I will resolve it recently. > The cast operation may push down

[jira] [Updated] (SPARK-27814) The cast operation may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27814: Summary: The cast operation may push down uncorrect filter, which is fatal. (was: The cast operation may

[jira] [Updated] (SPARK-27814) The cast operation for partitioned column may push down uncorrect filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27814: Summary: The cast operation for partitioned column may push down uncorrect filter, which is fatal. (was:

[jira] [Created] (SPARK-27814) The cast operation may push down an correct filter, which is fatal.

2019-05-23 Thread feiwang (JIRA)
feiwang created SPARK-27814: --- Summary: The cast operation may push down an correct filter, which is fatal. Key: SPARK-27814 URL: https://issues.apache.org/jira/browse/SPARK-27814 Project: Spark

[jira] [Created] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

2019-04-18 Thread feiwang (JIRA)
feiwang created SPARK-27515: --- Summary: [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired Key: SPARK-27515 URL: https://issues.apache.org/jira/browse/SPARK-27515

[jira] [Updated] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

2019-04-18 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27515: Description: When submit a spark yarn application, we first create a container launch context and store

[jira] [Updated] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

2019-04-18 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27515: Description: When submit a spark yarn application, we first create a container launch context and store

[jira] [Updated] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

2019-04-18 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27515: Description: When submit a spark yarn application, we first create a container launch context and store

[jira] [Updated] (SPARK-27515) [Deploy] When application master retry after a long time running, the hdfs delegation token may be expired

2019-04-18 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27515: Description: When submit a spark yarn application, we first create a container launch context and store

[jira] [Updated] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-05 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27637: Affects Version/s: 2.4.2 > If exception occured while fetching blocks by netty block transfer service,

[jira] [Created] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-05 Thread feiwang (JIRA)
feiwang created SPARK-27637: --- Summary: If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry Key: SPARK-27637 URL:

[jira] [Updated] (SPARK-27637) If exception occured while fetching blocks by netty block transfer service, check whether the relative executor is alive before retry

2019-05-05 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27637: Description: There are several kinds of shuffle client, blockTransferService and externalShuffleClient.

[jira] [Commented] (SPARK-28194) [SQL] A NoSuchElementException maybe thrown when EnsureRequirement

2019-07-04 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878664#comment-16878664 ] feiwang commented on SPARK-28194: - {code:java} create table if not exists test.nga(ac1 string, ac2

[jira] [Updated] (SPARK-28194) [SQL] A NoSuchElementException maybe thrown when EnsureRequirement

2019-06-27 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-28194: Description: {code:java} java.util.NoSuchElementException: None.get at

[jira] [Updated] (SPARK-28194) [SQL] A NoSuchElementException maybe thrown when EnsureRequirement

2019-06-27 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-28194: Summary: [SQL] A NoSuchElementException maybe thrown when EnsureRequirement (was: a

[jira] [Created] (SPARK-28194) a NoSuchElementException maybe thrown when EnsureRequirement

2019-06-27 Thread feiwang (JIRA)
feiwang created SPARK-28194: --- Summary: a NoSuchElementException maybe thrown when EnsureRequirement Key: SPARK-28194 URL: https://issues.apache.org/jira/browse/SPARK-28194 Project: Spark Issue

[jira] [Created] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
feiwang created SPARK-27562: --- Summary: Complete the verification mechanism for shuffle transmitted data Key: SPARK-27562 URL: https://issues.apache.org/jira/browse/SPARK-27562 Project: Spark

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: I will update this lately. > Complete the verification mechanism for shuffle transmitted

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: We've seen some shuffle data corruption during shuffle read phase. As described in

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: SPARK-27562 We've seen some shuffle data corruption during shuffle read phase. As

[jira] [Updated] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-24 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27562: Description: We've seen some shuffle data corruption during shuffle read phase. As described in

[jira] [Commented] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-25 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826602#comment-16826602 ] feiwang commented on SPARK-27562: - [~irashid] > Complete the verification mechanism for shuffle

[jira] [Comment Edited] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-25 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826602#comment-16826602 ] feiwang edited comment on SPARK-27562 at 4/26/19 2:57 AM: -- [~irashid] What is

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Description: When we set spark.sql.decimalOperations.allowPrecisionLoss=false. For the sql below, the

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Description: When we set spark.sql.decimalOperations.allowPrecisionLoss=false. For the sql below, the

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Attachment: screenshot-1.png > [SQL] Decimal precision overflow when don't allow precision loss >

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Description: When we set spark.sql.decimalOperations.allowPrecisionLoss=false. For the sql below, the

[jira] [Created] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
feiwang created SPARK-29000: --- Summary: [SQL] Decimal precision overflow when don't allow precision loss Key: SPARK-29000 URL: https://issues.apache.org/jira/browse/SPARK-29000 Project: Spark

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-05 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Description: When we set spark.sql.decimalOperations.allowPrecisionLoss=false. For the sql below, the

[jira] [Updated] (SPARK-29010) [SQL] Cast Decimal to double may lost precision

2019-09-06 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29010: Description: Even I set spark.sql.decimalOperations.allowPrecisionLoss=false. It will still lost

[jira] [Updated] (SPARK-29010) [SQL] Cast Decimal to double may lost precision

2019-09-06 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29010: Attachment: screenshot-1.png > [SQL] Cast Decimal to double may lost precision >

[jira] [Created] (SPARK-29010) [SQL] Cast Decimal to double may lost precision

2019-09-06 Thread feiwang (Jira)
feiwang created SPARK-29010: --- Summary: [SQL] Cast Decimal to double may lost precision Key: SPARK-29010 URL: https://issues.apache.org/jira/browse/SPARK-29010 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-29010) [SQL] Cast Decimal to double may lost precision

2019-09-06 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang resolved SPARK-29010. - Resolution: Not A Bug > [SQL] Cast Decimal to double may lost precision >

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928168#comment-16928168 ] feiwang commented on SPARK-29037: - This committedTaskPath is hard coded in FileOutputCommitter class. >

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928170#comment-16928170 ] feiwang commented on SPARK-29037: - If we have several applications, which insert overwrite a partition

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928165#comment-16928165 ] feiwang commented on SPARK-29037: - This is the unit test log. !screenshot-1.png! We can see that, the

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Attachment: screenshot-1.png > [Core] Spark gives duplicate result when an application was killed and

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Issue Comment Deleted] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Comment: was deleted (was: If we have several applications, which insert overwrite a partition of same

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928180#comment-16928180 ] feiwang commented on SPARK-29037: - [~cloud_fan] > [Core] Spark gives duplicate result when an

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Affects Version/s: 2.3.3 > [Core] Spark gives duplicate result when an application was killed and rerun >

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Updated] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Description: When we insert overwrite a partition of table. For a stage, whose tasks commit output, a

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928375#comment-16928375 ] feiwang commented on SPARK-29037: - If we set

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928394#comment-16928394 ] feiwang commented on SPARK-29037: - But for the version 2, it may produce partial result when we kill an

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928596#comment-16928596 ] feiwang commented on SPARK-29037: - [~advancedxy] 1. We re-submit the same application again. We meet

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928656#comment-16928656 ] feiwang commented on SPARK-29037: - In detail, I think we need change the logic of

[jira] [Issue Comment Deleted] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29037: Comment: was deleted (was: In detail, I think we need change the logic of

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928598#comment-16928598 ] feiwang commented on SPARK-29037: - The implementation of InsertIntoHiveTable prevent reuse same

[jira] [Comment Edited] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928596#comment-16928596 ] feiwang edited comment on SPARK-29037 at 9/12/19 2:39 PM: -- [~advancedxy] 1.

[jira] [Comment Edited] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928674#comment-16928674 ] feiwang edited comment on SPARK-29037 at 9/12/19 3:49 PM: -- [~advancedxy] Thanks

[jira] [Comment Edited] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928596#comment-16928596 ] feiwang edited comment on SPARK-29037 at 9/12/19 4:17 PM: -- [~advancedxy] 1.

[jira] [Comment Edited] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928674#comment-16928674 ] feiwang edited comment on SPARK-29037 at 9/12/19 3:48 PM: -- [~advancedxy] I just

[jira] [Comment Edited] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2019-09-12 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928674#comment-16928674 ] feiwang edited comment on SPARK-29037 at 9/12/19 3:58 PM: -- [~advancedxy] Thanks

  1   2   3   >