[
https://issues.apache.org/jira/browse/SPARK-41665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Prelle updated SPARK-41665:
----------------------------------
Description:
Hi,
We detect a strange behavior on spark streaming when we set a trigger interval
for example at 1 minutes all query will start at 0:00:00 0:01:00 0:02:00 no
matter the start time of the query.
So all query are "sync", so it's can disturbed a cluster a cluster i do leads
to spike of utilisation
!image-2022-12-21-07-57-18-679.png!
For me the expected behavior should be like this
!image-2022-12-21-07-57-32-654.png!
It's because of this line
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala#L98]
as now in intervalMS are long now / intervalMs * intervalMs will just cut in my
case the second, as it's explicitely like this on the test
([https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ProcessingTimeExecutorSuite.scala#L36)]
i do not know if it's the expected behavior or it's juste because this line
it's here since 6 years. So it's affecting all versions since 6 years.
Regards
Thomas
was:
Hi,
We detect a strange behavior on spark streaming when we set a trigger interval
for example at 1 minutes all query will start at 0:00:00 0:01:00 0:02:00 no
matter the start time of the query.
So all query are "sync", so it's can disturbed a cluster a cluster i do leads
to spike of utilisation
!image-2022-12-21-07-57-18-679.png!
For me the expected behavior should be like this
!image-2022-12-21-07-57-32-654.png!
It's because of this line
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala#L98]
as now in intervalMS are long now / intervalMs * intervalMs will just cut in my
case the second, as it's explicitely like this i do not know if it's the
expected behavior or it's juste because this line it's here since 6 years. So
it's affecting all versions since 6 years.
Regards
Thomas
> Spark streaming query scheduling synchronisation with Trigger Interval
> -----------------------------------------------------------------------
>
> Key: SPARK-41665
> URL: https://issues.apache.org/jira/browse/SPARK-41665
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.8, 3.0.3, 3.1.2, 3.2.2, 3.3.1
> Reporter: Thomas Prelle
> Priority: Major
> Attachments: image-2022-12-21-07-57-18-679.png,
> image-2022-12-21-07-57-32-654.png
>
>
> Hi,
> We detect a strange behavior on spark streaming when we set a trigger
> interval for example at 1 minutes all query will start at 0:00:00 0:01:00
> 0:02:00 no matter the start time of the query.
> So all query are "sync", so it's can disturbed a cluster a cluster i do leads
> to spike of utilisation
> !image-2022-12-21-07-57-18-679.png!
> For me the expected behavior should be like this
> !image-2022-12-21-07-57-32-654.png!
>
> It's because of this line
> [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala#L98]
> as now in intervalMS are long now / intervalMs * intervalMs will just cut in
> my case the second, as it's explicitely like this on the test
> ([https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ProcessingTimeExecutorSuite.scala#L36)]
> i do not know if it's the expected behavior or it's juste because this line
> it's here since 6 years. So it's affecting all versions since 6 years.
> Regards
> Thomas
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
