[jira] [Commented] (FLINK-29737) Support DataGen on waveform function
[ https://issues.apache.org/jira/browse/FLINK-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624376#comment-17624376 ] yuzelin commented on FLINK-29737: - Thanks for your idea. I have some points about this feature. # As far as I'm concerned, the `datagen` connector is usually used in test cases, so how the data are generated is usually ignored. I wonder whether you can share more details of `wave form generation rate` usage scenarios? I think maybe I need to discuss with [~fsk119] and other community members to estimate whether to support this feature. # To add support for sin-based waveform generation rate is not scalable enough. In my opinion, a better way is to introduce an option like `RateShape`[1], and mark current mode as `square`. It backs to point 1 that whether it's generic enough to introduce this new feature. Anyway, thanks again. [~fsk119] Please take a look. [1][`RateShape` example|https://github.com/nexmark/nexmark/blob/54974ef36a0d01ef8ebc0b4ba39cfc50136af0f6/nexmark-flink/src/main/java/com/github/nexmark/flink/utils/NexmarkUtils.java#L53] > Support DataGen on waveform function > > > Key: FLINK-29737 > URL: https://issues.apache.org/jira/browse/FLINK-29737 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: chenzihao >Priority: Major > Labels: pull-request-available > Attachments: image-2022-10-24-16-09-47-386.png, > image-2022-10-24-16-09-52-410.png > > > In some scenarios, we need to simulate flow changes in the production > environment. The current DATAGEN feature only supports data generation at a > constant rate. We try to simulate increments of flow using batch jobs, but > the production rate is not smooth, so I suggest that we can support sin-based > data generation in order to get smooth changes. > 1. add another batch job to simulate increments of flow. > !image-2022-10-24-16-09-52-410.png! > 2. sin-based. > !image-2022-10-24-16-09-47-386.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29737) Support DataGen on waveform function
[ https://issues.apache.org/jira/browse/FLINK-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624134#comment-17624134 ] Shengkai Fang commented on FLINK-29737: --- Sure. I have added this in my todo list. I will take a look when I am free. > Support DataGen on waveform function > > > Key: FLINK-29737 > URL: https://issues.apache.org/jira/browse/FLINK-29737 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: chenzihao >Priority: Major > Labels: pull-request-available > Attachments: image-2022-10-24-16-09-47-386.png, > image-2022-10-24-16-09-52-410.png > > > In some scenarios, we need to simulate flow changes in the production > environment. The current DATAGEN feature only supports data generation at a > constant rate. We try to simulate increments of flow using batch jobs, but > the production rate is not smooth, so I suggest that we can support sin-based > data generation in order to get smooth changes. > 1. add another batch job to simulate increments of flow. > !image-2022-10-24-16-09-52-410.png! > 2. sin-based. > !image-2022-10-24-16-09-47-386.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29737) Support DataGen on waveform function
[ https://issues.apache.org/jira/browse/FLINK-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623546#comment-17623546 ] chenzihao commented on FLINK-29737: --- [~fsk119] Thanks for your reply. I have opened the PR, please help to review it, thanks a lot. > Support DataGen on waveform function > > > Key: FLINK-29737 > URL: https://issues.apache.org/jira/browse/FLINK-29737 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: chenzihao >Priority: Major > Labels: pull-request-available > Attachments: image-2022-10-24-16-09-47-386.png, > image-2022-10-24-16-09-52-410.png > > > In some scenarios, we need to simulate flow changes in the production > environment. The current DATAGEN feature only supports data generation at a > constant rate. We try to simulate increments of flow using batch jobs, but > the production rate is not smooth, so I suggest that we can support sin-based > data generation in order to get smooth changes. > 1. add another batch job to simulate increments of flow. > !image-2022-10-24-16-09-52-410.png! > 2. sin-based. > !image-2022-10-24-16-09-47-386.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29737) Support DataGen on waveform function
[ https://issues.apache.org/jira/browse/FLINK-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623501#comment-17623501 ] Shengkai Fang commented on FLINK-29737: --- Thanks for your idea. Could you share the PR link here? > Support DataGen on waveform function > > > Key: FLINK-29737 > URL: https://issues.apache.org/jira/browse/FLINK-29737 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: chenzihao >Priority: Major > Attachments: image-2022-10-24-16-09-47-386.png, > image-2022-10-24-16-09-52-410.png > > > In some scenarios, we need to simulate flow changes in the production > environment. The current DATAGEN feature only supports data generation at a > constant rate. We try to simulate increments of flow using batch jobs, but > the production rate is not smooth, so I suggest that we can support sin-based > data generation in order to get smooth changes. > 1. add another batch job to simulate increments of flow. > !image-2022-10-24-16-09-52-410.png! > 2. sin-based. > !image-2022-10-24-16-09-47-386.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29737) Support DataGen on waveform function
[ https://issues.apache.org/jira/browse/FLINK-29737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623041#comment-17623041 ] chenzihao commented on FLINK-29737: --- I have simply extended the functionality of DATAGEN to support the smooth change of data generation rate. And support adjusting the data change period and the peak value. If this small improvement is useful for Flink, I am willing to open a PR. [~lzljs3620320] Please help to review this, thanks a lot. > Support DataGen on waveform function > > > Key: FLINK-29737 > URL: https://issues.apache.org/jira/browse/FLINK-29737 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: chenzihao >Priority: Major > Attachments: image-2022-10-24-16-09-47-386.png, > image-2022-10-24-16-09-52-410.png > > > In some scenarios, we need to simulate flow changes in the production > environment. The current DATAGEN feature only supports data generation at a > constant rate. We try to simulate increments of flow using batch jobs, but > the production rate is not smooth, so I suggest that we can support sin-based > data generation in order to get smooth changes. > 1. add another batch job to simulate increments of flow. > !image-2022-10-24-16-09-52-410.png! > 2. sin-based. > !image-2022-10-24-16-09-47-386.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
