[jira] [Reopened] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-19013: -- > java.util.ConcurrentModificationException when using s3 path as > checkpointLocation >

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799501#comment-15799501 ] Shixiong Zhu commented on SPARK-19013: -- [~zzztimbo] Could you provide the full stack trace so that I

[jira] [Updated] (SPARK-19036) Merging dealyed micro batches

[ https://issues.apache.org/jira/browse/SPARK-19036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19036: - Component/s: (was: Structured Streaming) DStreams > Merging dealyed micro

[jira] [Updated] (SPARK-19041) Fix code snippet compilation issues in Structured Streaming Programming Guide

[ https://issues.apache.org/jira/browse/SPARK-19041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19041: - Fix Version/s: 2.1.1 > Fix code snippet compilation issues in Structured Streaming Programming

[jira] [Resolved] (SPARK-19050) Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'

[ https://issues.apache.org/jira/browse/SPARK-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19050. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Fix

[jira] [Updated] (SPARK-19050) Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'

[ https://issues.apache.org/jira/browse/SPARK-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19050: - Component/s: Structured Streaming > Fix EventTimeWatermarkSuite 'delay in months and years

[jira] [Commented] (SPARK-19050) Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'

[ https://issues.apache.org/jira/browse/SPARK-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15791581#comment-15791581 ] Shixiong Zhu commented on SPARK-19050: -- monthsSinceEpoch in this test is like math.floor(num), so

[jira] [Created] (SPARK-19050) Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'

Shixiong Zhu created SPARK-19050: Summary: Fix EventTimeWatermarkSuite 'delay in months and years handled correctly' Key: SPARK-19050 URL: https://issues.apache.org/jira/browse/SPARK-19050 Project:

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15788152#comment-15788152 ] Shixiong Zhu commented on SPARK-17463: -- [~sunil.rangwani] Could you provide the codes that use

[jira] [Resolved] (SPARK-17766) Write ahead log exception on a toy project

[ https://issues.apache.org/jira/browse/SPARK-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17766. -- Resolution: Duplicate This has been fixed in SPARK-18617 > Write ahead log exception on a toy

[jira] [Resolved] (SPARK-18669) Update Apache docs regard watermarking in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-18669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18669. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Update Apache docs

[jira] [Commented] (SPARK-18805) InternalMapWithStateDStream make java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786000#comment-15786000 ] Shixiong Zhu commented on SPARK-18805: -- @etienne That should be not an infinite loop. The time is

[jira] [Commented] (SPARK-18942) Support output operations for kinesis

[ https://issues.apache.org/jira/browse/SPARK-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785953#comment-15785953 ] Shixiong Zhu commented on SPARK-18942: -- Thanks for your prototype. Actually, you can just implement

[jira] [Commented] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785930#comment-15785930 ] Shixiong Zhu commented on SPARK-18974: -- Do you want to try Structured Streaming? Its

[jira] [Updated] (SPARK-18359) Let user specify locale in CSV parsing

[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18359: - Component/s: (was: Spark Core) > Let user specify locale in CSV parsing >

[jira] [Commented] (SPARK-18404) RPC call from executor to driver blocks when getting map output locations (Netty Only)

[ https://issues.apache.org/jira/browse/SPARK-18404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785914#comment-15785914 ] Shixiong Zhu commented on SPARK-18404: -- That's pretty weird. It's a blocking call for both netty and

[jira] [Updated] (SPARK-19012) CreateOrReplaceTempView throws org.apache.spark.sql.catalyst.parser.ParseException when viewName first char is numerical

[ https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19012: - Component/s: (was: Spark Core) SQL > CreateOrReplaceTempView throws >

[jira] [Updated] (SPARK-17999) Add getPreferredLocations for KafkaSourceRDD

[ https://issues.apache.org/jira/browse/SPARK-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17999: - Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog

[ https://issues.apache.org/jira/browse/SPARK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-15698: - Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

[ https://issues.apache.org/jira/browse/SPARK-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-16963: - Component/s: (was: DStreams) Structured Streaming > Change Source API so

[jira] [Updated] (SPARK-17153) [Structured streams] readStream ignores partition columns

[ https://issues.apache.org/jira/browse/SPARK-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17153: - Component/s: (was: DStreams) Structured Streaming > [Structured streams]

[jira] [Updated] (SPARK-17085) Documentation and actual code differs - Unsupported Operations

[ https://issues.apache.org/jira/browse/SPARK-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17085: - Component/s: (was: DStreams) Structured Streaming > Documentation and

[jira] [Updated] (SPARK-17475) HDFSMetadataLog should not leak CRC files

[ https://issues.apache.org/jira/browse/SPARK-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17475: - Component/s: (was: DStreams) Structured Streaming > HDFSMetadataLog should

[jira] [Updated] (SPARK-17513) StreamExecution should discard unneeded metadata

[ https://issues.apache.org/jira/browse/SPARK-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17513: - Component/s: (was: DStreams) Structured Streaming > StreamExecution should

[jira] [Updated] (SPARK-18152) CLONE - FileStreamSource should not track the list of seen files indefinitely

[ https://issues.apache.org/jira/browse/SPARK-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18152: - Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18030: - Component/s: (was: DStreams) Structured Streaming > Flaky test:

[jira] [Updated] (SPARK-18151) CLONE - MetadataLog should support purging old logs

[ https://issues.apache.org/jira/browse/SPARK-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18151: - Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-18153) CLONE - Ability to remove old metadata for structure streaming MetadataLog

[ https://issues.apache.org/jira/browse/SPARK-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18153: - Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-18156) CLONE - StreamExecution should discard unneeded metadata

[ https://issues.apache.org/jira/browse/SPARK-18156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18156: - Component/s: (was: DStreams) Structured Streaming > CLONE - StreamExecution

[jira] [Updated] (SPARK-18154) CLONE - Change Source API so that sources do not need to keep unbounded state

[ https://issues.apache.org/jira/browse/SPARK-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18154: - Component/s: (was: DStreams) Structured Streaming > CLONE - Change Source

[jira] [Commented] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781279#comment-15781279 ] Shixiong Zhu commented on SPARK-18974: -- Not sure about HDFS. But I just checked my local file system

[jira] [Updated] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18974: - Component/s: (was: Input/Output) > FileInputDStream could not detected files which moved to

[jira] [Resolved] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19013. -- Resolution: Not A Problem This is basically a S3 problem, not a Spark one. >

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781019#comment-15781019 ] Shixiong Zhu commented on SPARK-19013: -- This is probably because S3 negative cache. "a negative

[jira] [Commented] (SPARK-18991) Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

[ https://issues.apache.org/jira/browse/SPARK-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779002#comment-15779002 ] Shixiong Zhu commented on SPARK-18991: -- [~prashant_] FYI, this may also be necessary for your Kafka

[jira] [Resolved] (SPARK-17755) Master may ask a worker to launch an executor before the worker actually got the response of registration

[ https://issues.apache.org/jira/browse/SPARK-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17755. -- Resolution: Fixed Fix Version/s: 2.2.0 > Master may ask a worker to launch an executor

[jira] [Updated] (SPARK-18991) Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

[ https://issues.apache.org/jira/browse/SPARK-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18991: - Description: Right now `ContextCleaner.referenceBuffer` is ConcurrentLinkedQueue and the time

[jira] [Resolved] (SPARK-18991) Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

[ https://issues.apache.org/jira/browse/SPARK-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18991. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Change

[jira] [Created] (SPARK-18991) Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

Shixiong Zhu created SPARK-18991: Summary: Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster Key: SPARK-18991 URL: https://issues.apache.org/jira/browse/SPARK-18991

[jira] [Resolved] (SPARK-18972) Fix the netty thread names for RPC

[ https://issues.apache.org/jira/browse/SPARK-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18972. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 2.0.3

[jira] [Resolved] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

[ https://issues.apache.org/jira/browse/SPARK-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18985. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Add missing

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771304#comment-15771304 ] Shixiong Zhu commented on SPARK-18970: -- I see. FileStreamSource ignores FileNotFoundException when

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771221#comment-15771221 ] Shixiong Zhu commented on SPARK-18970: -- Where did you find the exception? If it's in the driver,

[jira] [Created] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

Shixiong Zhu created SPARK-18985: Summary: Add missing @InterfaceStability.Evolving for Structured Streaming APIs Key: SPARK-18985 URL: https://issues.apache.org/jira/browse/SPARK-18985 Project:

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771166#comment-15771166 ] Shixiong Zhu commented on SPARK-18970: -- Did the Spark task fail or not? Looks like the Spark task

[jira] [Updated] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

[ https://issues.apache.org/jira/browse/SPARK-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18031: - Fix Version/s: 2.0.3 > Flaky test: >

[jira] [Updated] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

[ https://issues.apache.org/jira/browse/SPARK-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18031: - Fix Version/s: 2.2.0 > Flaky test: >

[jira] [Resolved] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18908. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > It's hard for the

[jira] [Created] (SPARK-18972) Fix the netty thread names for RPC

Shixiong Zhu created SPARK-18972: Summary: Fix the netty thread names for RPC Key: SPARK-18972 URL: https://issues.apache.org/jira/browse/SPARK-18972 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-18971) Netty issue may cause the shuffle client hang

Shixiong Zhu created SPARK-18971: Summary: Netty issue may cause the shuffle client hang Key: SPARK-18971 URL: https://issues.apache.org/jira/browse/SPARK-18971 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-18954) Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window

[ https://issues.apache.org/jira/browse/SPARK-18954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18954. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 2.0.3

[jira] [Resolved] (SPARK-18894) Event time watermark delay threshold specified in months or years gives incorrect results

[ https://issues.apache.org/jira/browse/SPARK-18894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18894. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Event time watermark

[jira] [Created] (SPARK-18954) Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window

Shixiong Zhu created SPARK-18954: Summary: Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window Key: SPARK-18954 URL: https://issues.apache.org/jira/browse/SPARK-18954

[jira] [Resolved] (SPARK-18927) MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in conf

[ https://issues.apache.org/jira/browse/SPARK-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18927. -- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18820) Driver may send "LaunchTask" before executor receive "RegisteredExecutor"

[ https://issues.apache.org/jira/browse/SPARK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764995#comment-15764995 ] Shixiong Zhu commented on SPARK-18820: -- This was fixed in SPARK-13112. It's for 2.0.0 not backported

[jira] [Resolved] (SPARK-18820) Driver may send "LaunchTask" before executor receive "RegisteredExecutor"

[ https://issues.apache.org/jira/browse/SPARK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18820. -- Resolution: Duplicate > Driver may send "LaunchTask" before executor receive

[jira] [Assigned] (SPARK-17755) Master may ask a worker to launch an executor before the worker actually got the response of registration

[ https://issues.apache.org/jira/browse/SPARK-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-17755: Assignee: Shixiong Zhu > Master may ask a worker to launch an executor before the worker

[jira] [Updated] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18908: - Priority: Critical (was: Blocker) > It's hard for the user to see the failure if

[jira] [Assigned] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18908: Assignee: Shixiong Zhu > It's hard for the user to see the failure if StreamExecution

[jira] [Updated] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18908: - Description: If the logical plan fails to create, e.g., some Source options are invalid, the

[jira] [Updated] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18908: - Description: If the logical plan fails to create, e.g., some options are invalid, the user

[jira] [Updated] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18908: - Priority: Blocker (was: Major) > It's hard for the user to see the failure if StreamExecution

[jira] [Created] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

Shixiong Zhu created SPARK-18908: Summary: It's hard for the user to see the failure if StreamExecution fails to create the logical plan Key: SPARK-18908 URL: https://issues.apache.org/jira/browse/SPARK-18908

[jira] [Updated] (SPARK-18908) It's hard for the user to see the failure if StreamExecution fails to create the logical plan

[ https://issues.apache.org/jira/browse/SPARK-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18908: - Affects Version/s: 2.1.0 > It's hard for the user to see the failure if StreamExecution fails to

[jira] [Updated] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18907: - Priority: Minor (was: Major) > Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max

[jira] [Created] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

Shixiong Zhu created SPARK-18907: Summary: Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values Key: SPARK-18907 URL:

[jira] [Assigned] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

[ https://issues.apache.org/jira/browse/SPARK-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18031: Assignee: Shixiong Zhu > Flaky test: >

[jira] [Resolved] (SPARK-18904) Merge two FileStreamSourceSuite files

[ https://issues.apache.org/jira/browse/SPARK-18904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18904. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Merge two

[jira] [Updated] (SPARK-18904) Merge two FileStreamSourceSuite files

[ https://issues.apache.org/jira/browse/SPARK-18904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18904: - Priority: Minor (was: Trivial) > Merge two FileStreamSourceSuite files >

[jira] [Updated] (SPARK-18904) Merge two FileStreamSourceSuite files

[ https://issues.apache.org/jira/browse/SPARK-18904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18904: - Priority: Trivial (was: Major) > Merge two FileStreamSourceSuite files >

[jira] [Created] (SPARK-18904) Merge two FileStreamSourceSuite files

Shixiong Zhu created SPARK-18904: Summary: Merge two FileStreamSourceSuite files Key: SPARK-18904 URL: https://issues.apache.org/jira/browse/SPARK-18904 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755189#comment-15755189 ] Shixiong Zhu commented on SPARK-18588: -- [~dongjoon] This commit just disabled the test. We still

[jira] [Updated] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18588: - Fix Version/s: (was: 2.1.0) > KafkaSourceStressForDontFailOnDataLossSuite is flaky >

[jira] [Reopened] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-18588: -- > KafkaSourceStressForDontFailOnDataLossSuite is flaky >

[jira] [Updated] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18588: - Target Version/s: (was: 2.1.0) > KafkaSourceStressForDontFailOnDataLossSuite is flaky >

[jira] [Resolved] (SPARK-18868) Flaky Test: StreamingQueryListenerSuite

[ https://issues.apache.org/jira/browse/SPARK-18868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18868. -- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-18850) Make StreamExecution and progress classes serializable

[ https://issues.apache.org/jira/browse/SPARK-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18850: - Target Version/s: 2.1.0 > Make StreamExecution and progress classes serializable >

[jira] [Updated] (SPARK-18850) Make StreamExecution and progress classes serializable

[ https://issues.apache.org/jira/browse/SPARK-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18850: - Affects Version/s: 2.1.0 > Make StreamExecution and progress classes serializable >

[jira] [Updated] (SPARK-18850) Make StreamExecution and progress classes serializable

[ https://issues.apache.org/jira/browse/SPARK-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18850: - Summary: Make StreamExecution and progress classes serializable (was: Make StreamExecution

[jira] [Updated] (SPARK-18850) Make StreamExecution and progress classes serializable

[ https://issues.apache.org/jira/browse/SPARK-18850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18850: - Description: Make StreamExecution and progress classes serializable because it is too easy for

[jira] [Resolved] (SPARK-18852) StreamingQuery.lastProgress should be null when recentProgress is empty

[ https://issues.apache.org/jira/browse/SPARK-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18852. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.1.0 >

[jira] [Created] (SPARK-18852) StreamingQuery.lastProgress should be null when recentProgress is empty

Shixiong Zhu created SPARK-18852: Summary: StreamingQuery.lastProgress should be null when recentProgress is empty Key: SPARK-18852 URL: https://issues.apache.org/jira/browse/SPARK-18852 Project:

[jira] [Updated] (SPARK-18852) StreamingQuery.lastProgress should be null when recentProgress is empty

[ https://issues.apache.org/jira/browse/SPARK-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18852: - Affects Version/s: 2.1.0 > StreamingQuery.lastProgress should be null when recentProgress is

[jira] [Resolved] (SPARK-18843) Fix timeout in awaitResultInForkJoinSafely

[ https://issues.apache.org/jira/browse/SPARK-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18843. -- Resolution: Fixed Fix Version/s: 2.1.1 2.0.3 > Fix timeout in

[jira] [Created] (SPARK-18850) Make StreamExecution serializable

Shixiong Zhu created SPARK-18850: Summary: Make StreamExecution serializable Key: SPARK-18850 URL: https://issues.apache.org/jira/browse/SPARK-18850 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746217#comment-15746217 ] Shixiong Zhu commented on SPARK-18838: -- [~sitalke...@gmail.com] Instead of your proposal, I prefer

[jira] [Created] (SPARK-18843) Fix timeout in awaitResultInForkJoinSafely

Shixiong Zhu created SPARK-18843: Summary: Fix timeout in awaitResultInForkJoinSafely Key: SPARK-18843 URL: https://issues.apache.org/jira/browse/SPARK-18843 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18773: - Assignee: Marcelo Vanzin > Make translation of Spark configs to commons-crypto configs

[jira] [Resolved] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18773. -- Resolution: Fixed > Make translation of Spark configs to commons-crypto configs consistent >

[jira] [Updated] (SPARK-18773) Make translation of Spark configs to commons-crypto configs consistent

[ https://issues.apache.org/jira/browse/SPARK-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18773: - Fix Version/s: 2.2.0 > Make translation of Spark configs to commons-crypto configs consistent >

[jira] [Resolved] (SPARK-18790) Keep a general offset history of stream batches

[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18790. -- Resolution: Fixed Assignee: Tyson Condie Fix Version/s: 2.1.1

[jira] [Created] (SPARK-18826) Make FileStream be able to start with most recent files

Shixiong Zhu created SPARK-18826: Summary: Make FileStream be able to start with most recent files Key: SPARK-18826 URL: https://issues.apache.org/jira/browse/SPARK-18826 Project: Spark

[jira] [Resolved] (SPARK-18811) Stream Source resolution should happen in StreamExecution thread, not main thread

[ https://issues.apache.org/jira/browse/SPARK-18811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18811. -- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-4105. - Resolution: Fixed Fix Version/s: 2.2.0 > FAILED_TO_UNCOMPRESS(5) errors when fetching

[jira] [Reopened] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-18588: -- It's still flaky > KafkaSourceStressForDontFailOnDataLossSuite is flaky >

[jira] [Reopened] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-13747: -- This issue still exists. Reopened it. > Concurrent execution in SQL doesn't work with Scala

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Component/s: Structured Streaming > StreamingQueryManager should not hold a lock when starting a

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Description: Otherwise, the user cannot start any queries when a query is starting. If a query

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Affects Version/s: 2.1.0 2.0.2 > StreamingQueryManager should not hold a

[jira] [Created] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

Shixiong Zhu created SPARK-18796: Summary: StreamingQueryManager should not hold a lock when starting a query Key: SPARK-18796 URL: https://issues.apache.org/jira/browse/SPARK-18796 Project: Spark

<    7   8   9   10   11   12   13   14   15   16   >