[jira] [Resolved] (SPARK-33596) NPE when there is no watermark metrics

2020-11-30 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu resolved SPARK-33596. --- Resolution: Won't Fix > NPE when there is no watermark metrics >

[jira] [Updated] (SPARK-33596) NPE when there is no watermark metrics

2020-11-30 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-33596: -- Summary: NPE when there is no watermark metrics (was: NPE when there is no EventTime) > NPE when

[jira] [Created] (SPARK-33596) NPE when there is no EventTime

2020-11-29 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-33596: - Summary: NPE when there is no EventTime Key: SPARK-33596 URL: https://issues.apache.org/jira/browse/SPARK-33596 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-31928) Flaky test: StreamingDeduplicationSuite.test no-data flag

2020-06-10 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130471#comment-17130471 ] Genmao Yu commented on SPARK-31928: --- There is a related pr: 

[jira] [Created] (SPARK-31953) Add Spark Structured Streaming History Server Support

2020-06-10 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-31953: - Summary: Add Spark Structured Streaming History Server Support Key: SPARK-31953 URL: https://issues.apache.org/jira/browse/SPARK-31953 Project: Spark Issue Type:

[jira] [Created] (SPARK-31913) StackOverflowError in FileScanRDD

2020-06-05 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-31913: - Summary: StackOverflowError in FileScanRDD Key: SPARK-31913 URL: https://issues.apache.org/jira/browse/SPARK-31913 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-31677) Use KVStore to cache stream query progress

2020-05-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-31677: -- Description: 1. Streaming query progress information are cached twice in *StreamExecution* and

[jira] [Updated] (SPARK-31677) Use KVStore to cache stream query progress

2020-05-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-31677: -- Environment: (was: 1. Streaming query progress information are cached twice in *StreamExecution*

[jira] [Created] (SPARK-31677) Use KVStore to cache stream query progress

2020-05-11 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-31677: - Summary: Use KVStore to cache stream query progress Key: SPARK-31677 URL: https://issues.apache.org/jira/browse/SPARK-31677 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-31593) Remove unnecessary streaming query progress update

2020-04-28 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-31593: - Summary: Remove unnecessary streaming query progress update Key: SPARK-31593 URL: https://issues.apache.org/jira/browse/SPARK-31593 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29973) Use nano time to calculate 'processedRowsPerSecond' to avoid 'NaN'/'Infinity'

2019-11-20 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-29973: - Summary: Use nano time to calculate 'processedRowsPerSecond' to avoid 'NaN'/'Infinity' Key: SPARK-29973 URL: https://issues.apache.org/jira/browse/SPARK-29973 Project:

[jira] [Updated] (SPARK-29683) Job failed due to executor failures all available nodes are blacklisted

2019-10-31 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-29683: -- Description: My streaming job will fail *due to executor failures all available nodes are

[jira] [Created] (SPARK-29683) Job failed due to executor failures all available nodes are blacklisted

2019-10-31 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-29683: - Summary: Job failed due to executor failures all available nodes are blacklisted Key: SPARK-29683 URL: https://issues.apache.org/jira/browse/SPARK-29683 Project: Spark

[jira] [Created] (SPARK-29543) Support Structured Streaming UI

2019-10-21 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-29543: - Summary: Support Structured Streaming UI Key: SPARK-29543 URL: https://issues.apache.org/jira/browse/SPARK-29543 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-29438: -- Description: Now, Spark use the `TaskPartitionId` to determine the StateStore path. {code:java}

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949461#comment-16949461 ] Genmao Yu commented on SPARK-29438: --- [~kabhwan] Yes, the whole statestore path is

[jira] [Comment Edited] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu edited comment on SPARK-29438 at 10/11/19 11:15 AM: -- There are

[jira] [Comment Edited] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu edited comment on SPARK-29438 at 10/11/19 11:15 AM: -- There are

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu commented on SPARK-29438: --- There are several optional alternatives to resolve this issue: *

[jira] [Updated] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-29438: -- Description: Now, Spark use the `TaskPartitionId` to determine the StateStore path. {code:java}

[jira] [Created] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-29438: - Summary: Failed to get state store in stream-stream join Key: SPARK-29438 URL: https://issues.apache.org/jira/browse/SPARK-29438 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28256) Failed to initialize FileContextBasedCheckpointFileManager with uri without authority

2019-07-05 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-28256: -- Description: reproduce code {code:sql} CREATE TABLE `user_click_count` (`userId` STRING, `click`

[jira] [Updated] (SPARK-28256) Failed to initialize FileContextBasedCheckpointFileManager with uri without authority

2019-07-05 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-28256: -- Description: {code:java} java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

[jira] [Created] (SPARK-28256) Failed to initialize FileContextBasedCheckpointFileManager with uri without authority

2019-07-05 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-28256: - Summary: Failed to initialize FileContextBasedCheckpointFileManager with uri without authority Key: SPARK-28256 URL: https://issues.apache.org/jira/browse/SPARK-28256

[jira] [Created] (SPARK-28158) Hive UDFs supports UDT type

2019-06-25 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-28158: - Summary: Hive UDFs supports UDT type Key: SPARK-28158 URL: https://issues.apache.org/jira/browse/SPARK-28158 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27717) support UNION in continuous processing

2019-05-15 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-27717: - Summary: support UNION in continuous processing Key: SPARK-27717 URL: https://issues.apache.org/jira/browse/SPARK-27717 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-27715) SQL query details in UI dose not show in correct format.

2019-05-15 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-27715: - Summary: SQL query details in UI dose not show in correct format. Key: SPARK-27715 URL: https://issues.apache.org/jira/browse/SPARK-27715 Project: Spark Issue

[jira] [Comment Edited] (SPARK-26302) retainedBatches configuration can eat up memory on driver

2019-05-14 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839253#comment-16839253 ] Genmao Yu edited comment on SPARK-26302 at 5/14/19 9:27 AM: Adding some

[jira] [Commented] (SPARK-26302) retainedBatches configuration can eat up memory on driver

2019-05-14 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839253#comment-16839253 ] Genmao Yu commented on SPARK-26302: --- Add some warning in documentation is reasonable. >

[jira] [Commented] (SPARK-26278) V2 Streaming sources cannot be written to V1 sinks

2019-05-14 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839246#comment-16839246 ] Genmao Yu commented on SPARK-26278: --- [~jpolchlo] Could you please close this jira? This issue has been

[jira] [Commented] (SPARK-27634) deleteCheckpointOnStop should be configurable

2019-05-05 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833281#comment-16833281 ] Genmao Yu commented on SPARK-27634: --- Do not add patch here. You can submit a PR to Spark. [How to

[jira] [Created] (SPARK-27503) JobGenerator thread exit for some fatal errors but application keeps running

2019-04-18 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-27503: - Summary: JobGenerator thread exit for some fatal errors but application keeps running Key: SPARK-27503 URL: https://issues.apache.org/jira/browse/SPARK-27503 Project:

[jira] [Created] (SPARK-27413) Keep the same epoch pace between driver and executor.

2019-04-09 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-27413: - Summary: Keep the same epoch pace between driver and executor. Key: SPARK-27413 URL: https://issues.apache.org/jira/browse/SPARK-27413 Project: Spark Issue Type:

[jira] [Created] (SPARK-27355) make query execution more sensitive to epoch message late or lost

2019-04-03 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-27355: - Summary: make query execution more sensitive to epoch message late or lost Key: SPARK-27355 URL: https://issues.apache.org/jira/browse/SPARK-27355 Project: Spark

[jira] [Commented] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-21 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798635#comment-16798635 ] Genmao Yu commented on SPARK-27218: --- Could you please test it on master branch? >

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-08-23 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591151#comment-16591151 ] Genmao Yu commented on SPARK-24630: --- [~Jackey Lee] I am glad to participate in code review.  > SPIP:

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-08-01 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565252#comment-16565252 ] Genmao Yu commented on SPARK-24630: --- [~Jackey Lee] Pretty good!  We also have the SQL Streaming

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu edited comment on SPARK-24630 at 7/30/18 11:27 AM: - Practice to

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: (was: image-2018-07-30-18-48-38-352.png) > SPIP: Support SQLStreaming in Spark >

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: (was: image-2018-07-30-18-06-30-506.png) > SPIP: Support SQLStreaming in Spark >

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu edited comment on SPARK-24630 at 7/30/18 10:53 AM: - Try to add the

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561762#comment-16561762 ] Genmao Yu commented on SPARK-24630: --- Try to add the StreamSQL DDL, like this:

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: image-2018-07-30-18-48-38-352.png > SPIP: Support SQLStreaming in Spark >

[jira] [Updated] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-24630: -- Attachment: image-2018-07-30-18-06-30-506.png > SPIP: Support SQLStreaming in Spark >

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-25 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556837#comment-16556837 ] Genmao Yu edited comment on SPARK-24630 at 7/26/18 3:24 AM: [~zsxwing] Is

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-25 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556837#comment-16556837 ] Genmao Yu commented on SPARK-24630: --- [~zsxwing] Is there plan to better support SQL on streaming? 

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554043#comment-16554043 ] Genmao Yu edited comment on SPARK-24630 at 7/24/18 11:38 AM: - [~zsxwing]

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554043#comment-16554043 ] Genmao Yu commented on SPARK-24630: --- {{Structured Streaming supports standard SQL as the batch

[jira] [Comment Edited] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-07-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554043#comment-16554043 ] Genmao Yu edited comment on SPARK-24630 at 7/24/18 10:07 AM: - {{Structured

[jira] [Created] (SPARK-20672) Keep the `isStreaming` property in triggerLogicalPlan in Structured Streaming

2017-05-09 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-20672: - Summary: Keep the `isStreaming` property in triggerLogicalPlan in Structured Streaming Key: SPARK-20672 URL: https://issues.apache.org/jira/browse/SPARK-20672 Project:

[jira] [Commented] (SPARK-20139) Spark UI reports partial success for completed stage while log shows all tasks are finished

2017-03-30 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950244#comment-15950244 ] Genmao Yu commented on SPARK-20139: --- The event queue exceeds its capacity, so new events will be

[jira] [Commented] (SPARK-20065) Empty output files created for aggregation query in append mode

2017-03-23 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937904#comment-15937904 ] Genmao Yu commented on SPARK-20065: --- Make sense, I will give a fast update. > Empty output files

[jira] [Commented] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-22 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937675#comment-15937675 ] Genmao Yu commented on SPARK-20061: --- Colon is not supported in hadoop, see

[jira] [Created] (SPARK-20021) Miss backslash in python code

2017-03-19 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-20021: - Summary: Miss backslash in python code Key: SPARK-20021 URL: https://issues.apache.org/jira/browse/SPARK-20021 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19926) Make pyspark exception more readable

2017-03-12 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19926: -- Description: Exception in pyspark is a little difficult to read. like: {code} Traceback (most recent

[jira] [Created] (SPARK-19926) Make pyspark exception more readable

2017-03-12 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19926: - Summary: Make pyspark exception more readable Key: SPARK-19926 URL: https://issues.apache.org/jira/browse/SPARK-19926 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19853) Uppercase Kafka topics fail when startingOffsets are SpecificOffsets

2017-03-08 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901158#comment-15901158 ] Genmao Yu commented on SPARK-19853: --- Good catch! I will open a pr to fix this. Could you please help to

[jira] [Created] (SPARK-19861) watermark should not be a negative time.

2017-03-07 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19861: - Summary: watermark should not be a negative time. Key: SPARK-19861 URL: https://issues.apache.org/jira/browse/SPARK-19861 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-19822) CheckpointSuite.testCheckpointedOperation: should not check checkpointFilesOfLatestTime by the PATH string.

2017-03-04 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19822: - Summary: CheckpointSuite.testCheckpointedOperation: should not check checkpointFilesOfLatestTime by the PATH string. Key: SPARK-19822 URL:

[jira] [Comment Edited] (SPARK-19807) Add reason for cancellation when a stage is killed using web UI

2017-03-03 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894185#comment-15894185 ] Genmao Yu edited comment on SPARK-19807 at 3/3/17 11:35 AM:

[jira] [Commented] (SPARK-19807) Add reason for cancellation when a stage is killed using web UI

2017-03-03 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894185#comment-15894185 ] Genmao Yu commented on SPARK-19807: ---

[jira] [Updated] (SPARK-19805) Log the row type when query result dose not match

2017-03-02 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19805: -- Summary: Log the row type when query result dose not match (was: Log the row type when query result

[jira] [Created] (SPARK-19805) Log the row type when query result dose match

2017-03-02 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19805: - Summary: Log the row type when query result dose match Key: SPARK-19805 URL: https://issues.apache.org/jira/browse/SPARK-19805 Project: Spark Issue Type:

[jira] [Closed] (SPARK-19349) Check resource ready to avoid multiple receivers to be scheduled on the same node.

2017-03-02 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu closed SPARK-19349. - Resolution: Won't Fix > Check resource ready to avoid multiple receivers to be scheduled on the same >

[jira] [Created] (SPARK-19800) Implement one kind of streaming sampling - reservoir sampling

2017-03-02 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19800: - Summary: Implement one kind of streaming sampling - reservoir sampling Key: SPARK-19800 URL: https://issues.apache.org/jira/browse/SPARK-19800 Project: Spark

[jira] [Commented] (SPARK-19738) Consider adding error handler to DataStreamWriter

2017-02-27 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887218#comment-15887218 ] Genmao Yu commented on SPARK-19738: --- [~jlalwani] I tested it on latest master branch, and return NULL

[jira] [Comment Edited] (SPARK-19738) Consider adding error handler to DataStreamWriter

2017-02-27 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885374#comment-15885374 ] Genmao Yu edited comment on SPARK-19738 at 2/27/17 9:44 AM: [~gaaldornick]

[jira] [Comment Edited] (SPARK-19738) Consider adding error handler to DataStreamWriter

2017-02-27 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885374#comment-15885374 ] Genmao Yu edited comment on SPARK-19738 at 2/27/17 9:45 AM: [~gaaldornick]

[jira] [Created] (SPARK-19749) Name socket source with a meaningful name

2017-02-27 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19749: - Summary: Name socket source with a meaningful name Key: SPARK-19749 URL: https://issues.apache.org/jira/browse/SPARK-19749 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19738) Consider adding error handler to DataStreamWriter

2017-02-27 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885374#comment-15885374 ] Genmao Yu commented on SPARK-19738: --- [~gaaldornick] Sorry I can not reproduce it. > Consider adding

[jira] [Issue Comment Deleted] (SPARK-19699) createOrReplaceTable does not always replace an existing table of the same name

2017-02-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19699: -- Comment: was deleted (was: Good catch! Maybe we can add {{rdd.id}} or something else. [~cloud_fan]

[jira] [Commented] (SPARK-19699) createOrReplaceTable does not always replace an existing table of the same name

2017-02-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882385#comment-15882385 ] Genmao Yu commented on SPARK-19699: --- Good catch! Maybe we can add {{rdd.id}} or something else.

[jira] [Comment Edited] (SPARK-19699) createOrReplaceTable does not always replace an existing table of the same name

2017-02-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882385#comment-15882385 ] Genmao Yu edited comment on SPARK-19699 at 2/24/17 10:16 AM: - Good catch!

[jira] [Closed] (SPARK-19642) Improve the security guarantee for rest api and ui

2017-02-21 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu closed SPARK-19642. - Resolution: Won't Fix > Improve the security guarantee for rest api and ui >

[jira] [Created] (SPARK-19676) Flaky test: FsHistoryProviderSuite.SPARK-3697: ignore directories that cannot be read.

2017-02-21 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19676: - Summary: Flaky test: FsHistoryProviderSuite.SPARK-3697: ignore directories that cannot be read. Key: SPARK-19676 URL: https://issues.apache.org/jira/browse/SPARK-19676

[jira] [Updated] (SPARK-19642) Improve the security guarantee for rest api and ui

2017-02-16 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19642: -- Summary: Improve the security guarantee for rest api and ui (was: Improve the security guarantee for

[jira] [Updated] (SPARK-19642) Improve the security guarantee for rest api

2017-02-16 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19642: -- Description: As Spark gets more and more features, data may start leaking through other places (e.g.

[jira] [Updated] (SPARK-19642) Improve the security guarantee for rest api

2017-02-16 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-19642: -- Description: As Spark gets more and more features, data may start leaking through other places (e.g.

[jira] [Commented] (SPARK-19642) Improve the security guarantee for rest api

2017-02-16 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871058#comment-15871058 ] Genmao Yu commented on SPARK-19642: --- cc [~ajbozarth], [~vanzin] and [~srowen] > Improve the security

[jira] [Created] (SPARK-19642) Improve the security guarantee for rest api

2017-02-16 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19642: - Summary: Improve the security guarantee for rest api Key: SPARK-19642 URL: https://issues.apache.org/jira/browse/SPARK-19642 Project: Spark Issue Type:

[jira] [Created] (SPARK-19605) Fail it if existing resource is not enough to run streaming job

2017-02-14 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19605: - Summary: Fail it if existing resource is not enough to run streaming job Key: SPARK-19605 URL: https://issues.apache.org/jira/browse/SPARK-19605 Project: Spark

[jira] [Commented] (SPARK-19556) Broadcast data is not encrypted when I/O encryption is on

2017-02-14 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867152#comment-15867152 ] Genmao Yu commented on SPARK-19556: --- [~vanzin] I am working on this, could you please assign it to me?

[jira] [Commented] (SPARK-19524) newFilesOnly does not work according to docs.

2017-02-08 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858819#comment-15858819 ] Genmao Yu commented on SPARK-19524: --- Current implementation will clear the old time-to-files mappings

[jira] [Created] (SPARK-19482) Fail it if 'spark.master' is set with different value

2017-02-06 Thread Genmao Yu (JIRA)
Genmao Yu created SPARK-19482: - Summary: Fail it if 'spark.master' is set with different value Key: SPARK-19482 URL: https://issues.apache.org/jira/browse/SPARK-19482 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19451) Long values in Window function

2017-02-06 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853773#comment-15853773 ] Genmao Yu commented on SPARK-19451: --- [~jchamp] I have taken a fast look through the code, and did not

[jira] [Comment Edited] (SPARK-19451) Long values in Window function

2017-02-06 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853773#comment-15853773 ] Genmao Yu edited comment on SPARK-19451 at 2/6/17 9:58 AM: --- [~jchamp] I have

[jira] [Commented] (SPARK-19451) Long values in Window function

2017-02-05 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853434#comment-15853434 ] Genmao Yu commented on SPARK-19451: --- Good catch! I will dig deeply into code and fix it if it is really

[jira] [Commented] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-02-05 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853426#comment-15853426 ] Genmao Yu commented on SPARK-19407: --- [~aassudani] Are you still working on this? As this issue is clear

[jira] [Commented] (SPARK-19147) netty throw NPE

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837333#comment-15837333 ] Genmao Yu commented on SPARK-19147: --- After dig into code, this issue may occurs when executor is

[jira] [Comment Edited] (SPARK-19147) netty throw NPE

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837333#comment-15837333 ] Genmao Yu edited comment on SPARK-19147 at 1/25/17 7:39 AM: After dig into

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837048#comment-15837048 ] Genmao Yu commented on SPARK-19354: --- IMHO, the killed tasks will be failed finally, so there is no

[jira] [Commented] (SPARK-10141) Number of tasks on executors still become negative after failures

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837043#comment-15837043 ] Genmao Yu commented on SPARK-10141: --- I think this is fix in https://github.com/apache/spark/pull/14969,

[jira] [Commented] (SPARK-19356) Number of active tasks is negative even when there is no failed executor

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837041#comment-15837041 ] Genmao Yu commented on SPARK-19356: --- I think this is fix in https://github.com/apache/spark/pull/14969,

[jira] [Issue Comment Deleted] (SPARK-18563) mapWithState: initialState should have a timeout setting per record

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-18563: -- Comment: was deleted (was: I do not know is there any plan to add new feature to DStreams? Maybe, we

[jira] [Closed] (SPARK-19343) Do once optimistic checkpoint before stop

2017-01-24 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu closed SPARK-19343. - Resolution: Won't Fix > Do once optimistic checkpoint before stop >

[jira] [Comment Edited] (SPARK-18563) mapWithState: initialState should have a timeout setting per record

2017-01-22 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833416#comment-15833416 ] Genmao Yu edited comment on SPARK-18563 at 1/22/17 9:35 AM: I do not know is

[jira] [Commented] (SPARK-18563) mapWithState: initialState should have a timeout setting per record

2017-01-22 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833416#comment-15833416 ] Genmao Yu commented on SPARK-18563: --- I do not know is there any plan to add new feature to DStreams?

[jira] [Commented] (SPARK-18839) Executor is active on web, but actually is dead

2017-01-22 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833409#comment-15833409 ] Genmao Yu commented on SPARK-18839: --- Sorry, I do not think this is a bug. > Executor is active on web,

[jira] [Commented] (SPARK-18805) InternalMapWithStateDStream make java.lang.StackOverflowError

2017-01-22 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833377#comment-15833377 ] Genmao Yu commented on SPARK-18805: --- + 1 to {{That should be not an infinite loop. The time is

[jira] [Updated] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-20 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-18116: -- Target Version/s: (was: 2.0.3, 2.1.1) Fix Version/s: (was: 2.1.1)

  1   2   >