退订
[jira] [Created] (FLINK-32644) ts
weitianpei created FLINK-32644: -- Summary: ts Key: FLINK-32644 URL: https://issues.apache.org/jira/browse/FLINK-32644 Project: Flink Issue Type: Bug Reporter: weitianpei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32643) Introduce off-heap shared state cache across stateful operators in TM
Fang Yong created FLINK-32643: - Summary: Introduce off-heap shared state cache across stateful operators in TM Key: FLINK-32643 URL: https://issues.apache.org/jira/browse/FLINK-32643 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Affects Versions: 1.19.0 Reporter: Fang Yong Currently each stateful operator will create an independent db instance if it uses rocksdb as state backend, and we can configure `state.backend.rocksdb.block.cache-size` for each db to speed up state performance. This parameter defaults to 8M, and we cannot set it too large, such as 512M, this may cause OOM and each DB cannot effectively utilize memory. To address this issue, we would like to introduce off-heap shared state cache across multiple db instances for stateful operators in TM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Rocksdb info
Thanks guys! This was really helpful On Fri, 21 Jul 2023 at 8:49 AM, tison wrote: > > This is the case. > > Yanfei is correct that the uploading happens during the checkpoint stage. > > Before checkpointing, there is a snapshotting stage that synchronously > takes RocksDB snapshot by forcing all memtables into sstables on disk, and > hard-linked in a local temporary directory. > > In the checkpointing stage, RocksDB snapshot will be uploaded to OSS or DFS > fully or incrementally according to strategy. > > Best, > tison. > > > Yanfei Lei 于2023年7月21日周五 11:01写道： > > > Hi Taher, > > > > > is flink using rocks-cloud too? Or flink is moving rocks > > files after checkpoints? > > > > Flink isn't using rocks-cloud. flink would upload local rocks files to > > s3/DFS during checkpointing. Here is the related code: > > > > - RocksNativeFullSnapshotStrategy: > > > > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksNativeFullSnapshotStrategy.java#L186 > > - RocksIncrementalSnapshotStrategy: > > > > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L291 > > > > Taher Koitawala 于2023年7月21日周五 10:38写道： > > > > > > Hi all, > > > I know that Flink uses rocksdb to write checkpoints. While > > > researching rocksdb i found that there is something called rocks-cloud > > that > > > writes data over to s3. I know flink also does checkpoints over rocksdb > > to > > > s3. Question is, is flink using rocks-cloud too? Or flink is moving > rocks > > > files after checkpoints? > > > > > > Please share the code reference where this is done in flink. > > > > > > > > > Regards, > > > Taher Koitawala > > > > > > > > -- > > Best, > > Yanfei > > >
Re: Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
Hi Wencong, Thanks for the clarification. I got your point. It makes sense. Wrt IOReadableWritable, the suggestion was to check all classes that implemented it, e.g. BlockInfo, Value, Configuration, etc. Not limited to the Path. Best regards, Jing On Fri, Jul 21, 2023 at 4:31 PM Wencong Liu wrote: > Hello Jing, > > > Thanks for your reply. The URI field should be final and the > Path will be immutable.The static method deserializeFromDataInputView > will create a new Path object instead of replacing the URI field > in a existed Path Object. > > > For the crossing multiple modules issue, I've explained it in the reply > to Matthias. > > > Best regards, > > > Wencong Liu > > > > > > > > > > > > > > > > > > > > At 2023-07-21 18:05:26, "Jing Ge" wrote: > >Hi Wencong, > > > >Just out of curiosity, will the newly introduced > >deserializeFromDataInputView() method make the Path mutable again? > > > >What Matthias suggested makes sense, although the extension might make > this > >FLIP cross multiple modules. > > > >Best regards, > >Jing > > > >On Fri, Jul 21, 2023 at 10:23 AM Matthias Pohl > > wrote: > > > >> There's a kind-of-related issue FLINK-4758 [1] that proposes removing > the > >> IOReadableWritable interface from more classes. It was briefly > mentioned in > >> the must-have work items discussion [2]. > >> > >> I'm not too sure about the usage of IOReadableWritable: ...whether it > would > >> go away with the removal of the DataSet API in general (the Jira issue > has > >> DataSet as a component), anyway. > >> > >> Otherwise, might it make sense to extend the scope of this FLIP? > >> > >> [1] https://issues.apache.org/jira/browse/FLINK-4758 > >> [2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r > >> > >> On Fri, Jul 21, 2023 at 6:04 AM Xintong Song > >> wrote: > >> > >> > +1 > >> > > >> > Best, > >> > > >> > Xintong > >> > > >> > > >> > > >> > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu > >> wrote: > >> > > >> > > Hi devs, > >> > > > >> > > I would like to start a discussion on FLIP-347: Remove > >> IOReadableWritable > >> > > serialization in Path [1]. > >> > > > >> > > > >> > > The Path class is currently mutable to support IOReadableWritable > >> > > serialization. However, many parts > >> > > of the code assume that the Path is immutable. By making the Path > class > >> > > immutable, we can ensure > >> > > that paths are stored correctly without the possibility of mutation > and > >> > > eliminate the occurrence of subtle errors. > >> > > As such I propose to modify the Path class to no longer implement > the > >> > > IOReadableWritable interface. > >> > > Looking forward to your feedback. > >> > > [1] > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path > >> > > Best regards, > >> > > > >> > > > >> > > Wencong Liu > >> > > >> >
[jira] [Created] (FLINK-32642) The upsert mode doesn't work for the compound keys
jasonliangyc created FLINK-32642:
Summary: The upsert mode doesn't work for the compound keys
Key: FLINK-32642
URL: https://issues.apache.org/jira/browse/FLINK-32642
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Reporter: jasonliangyc
Attachments: image-2023-07-21-23-55-47-399.png,
image-2023-07-21-23-56-56-543.png, image-2023-07-21-23-57-22-186.png,
image-2023-07-22-00-11-56-363.png
Hi, the issue can be produced by following below steps:
*1.* Create two tables in sqlserver, one is the sink table, the other one is
cdc source table, the sink table has a compound key(date_str,time_str).
*2.* Create the corresponding flink tables in sql-client.
{code:java}
--create sink table in sqlserver
CREATE TABLE cumulative_cnt (
date_str VARCHAR(50),
time_str VARCHAR(50),
cnt INTEGER
CONSTRAINT PK_cumulative_cnt PRIMARY KEY (date_str,time_str)
);
--create source cdc table in sqlserver
CREATE TABLE user_behavior (
id INTEGER NOT NULL IDENTITY(101,1) PRIMARY KEY,
create_date datetime NOT NULL,
click_event VARCHAR(255) NOT NULL
);
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'user_behavior',
@role_name = NULL,
@supports_net_changes = 1
GO
--create flink tables through sql-client
CREATE TABLE cumulative_cnt (
date_str STRING,
time_str STRING,
cnt BIGINT,
PRIMARY KEY (date_str, time_str)NOT ENFORCED
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:sqlserver://:1433;databaseName=',
'username' = '',
'password' = '',
'table-name' = 'cumulative_cnt'
);
--create flink cdc table through sql-client
CREATE TABLE user_behavior (
id int,
create_date TIMESTAMP(0),
click_event STRING
) WITH (
'connector' = 'sqlserver-cdc',
'hostname' = '',
'port' = '1433',
'username' = '',
'password' = '',
'database-name' = '',
'schema-name' = 'dbo',
'table-name' = 'user_behavior'
); {code}
*3.* Run below sql through sql-client to start the job for capturing the cdc
data and do the aggregation and finally insert the result into target table.
{code:java}
insert into cumulative_cnt
select
date_str,
max(time_str) as time_str,
count(*) as cnt
from
(
select
DATE_FORMAT(create_date, '-MM-dd') as date_str,
SUBSTR(DATE_FORMAT(create_date, 'HH:mm'),1,4) || '0' as time_str
from user_behavior
) group by date_str;
{code}
*4.* Insert two records for testing.
{code:java}
INSERT INTO user_behavior(create_date, click_event)VALUES ('2023-06-01
01:01:00','click1');
INSERT INTO user_behavior(create_date, click_event)VALUES ('2023-06-01
02:20:00','click1');{code}
*5.* Checked the result in db( pls see the screen 1) and found that the target
table only have one record, but it is not the expectation cause the two source
records have different time, thus the compound key(date_str, time_str) shoud
be different( pls see the screen 2 )
There should be two records in the target table:
2023-06-01 01:00 1
2023-06-01 02:20 2
screen 1
!image-2023-07-22-00-11-56-363.png|width=274,height=199!
screen 2
!image-2023-07-21-23-57-22-186.png|width=572,height=120!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data
Hi Dong, Thanks a lot for the answers. I can now only briefly answer your last email. > It is possible that spilling to disks might cause larger overhead. IMO it > is an orthogonal issue already existing in Flink. This is because a Flink > job running batch mode might also be slower than its throughput in stream > mode due to the same reason. Yes, I know, but the thing that worries me is that previously only a user alone could decide whether to use batch mode or streaming, and in practice one user would rarely (if ever) use both for the same problem/job/query. If his intention was to eventually process live data, he was using streaming even if there was a large backlog at the start (apart of some very few very power users). With this change, we want to introduce a mode that would be switching back and forth between streaming and "batch in streaming" automatically. So a potential performance regression would be much more visible and painful at the same time. If batch query runs slower then it could, it's kind of fine as it will end at some point. If streaming query during large back pressure maybe temporary load spike switches to batch processing, that's a bigger deal. Especially if batch processing mode will not be able to actually even handle the normal load, after the load spike. In that case, the job could never recover from the backpressure/backlog mode. > execution.backlog.use-full-batch-mode-on-start (default false) ops sorry, it was supposed to be sth like: execution.backlog.use-batch-mode-only-on-start (default false) That option would disallow switching from streaming to batch. Batch mode would be allowed only to get rid of the initial, present on start-up backlog. Would allow us to safely experiment with switching from streaming to batch and I would be actually more fine in enabling "using batch mode on start" by default, until we gain confidence and feedback that switching back & forth is working as expected. >> Or we could limit the scope of this FLIP to only support starting with >> batch mode and switching only once to >> streaming, and design a follow up with switching back and forth? > > Sure, that sounds good to me. I am happy to split this FLIP into two FLIPs > so that we can make incremental progress. Great, let's do that. In a follow up FLIP we could restart the discussion about switching back and forth. Piotrek czw., 20 lip 2023 o 16:57 Dong Lin napisał(a): > Hi Piotr, > > Thank you for the very detailed comments! Please see my reply inline. > > On Thu, Jul 20, 2023 at 12:24 AM Piotr Nowojski > wrote: > > > Hi Dong, > > > > I have a couple of follow up questions about switching back and forth > > between streaming and batching mode. > > Especially around shuffle/watermark strategy, and keyed state backend. > > > > First of all, it might not always be beneficial to switch into the batch > > modes: > > - Shuffle strategy > > - Is sorting going to be purely in-memory? If not, obviously spilling > > to disks might cause larger overheads > >compared to not sorting the records. > > > > Sorting might require spilling data to disk depending on the input size. > The behavior of sorting w.r.t. memory/disk is expected to be exactly the > same as the behavior of input sorting automatically performed by Flink > runtime in batch mode for keyed inputs. > > More specifically, ExternalSorter > < > https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/ExternalSorter.java > > > is > currently used to sort keyed inputs in batch mode. It is automatically used > by Flink runtime in OneInputStreamTask (here > < > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OneInputStreamTask.java#L114 > >) > and in MultiInputSortingDataInput (here > < > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java#L188 > >). > We plan to re-use the same code/mechanism to do sorting. > > It is possible that spilling to disks might cause larger overhead. IMO it > is an orthogonal issue already existing in Flink. This is because a Flink > job running batch mode might also be slower than its throughput in stream > mode due to the same reason. However, even though it is possible in theory, > I expect that in practice the throughput of using sorting + > BatchExecutionKeyedStateBackend should be much higher than using other > keyed statebackends when the amount of data is large. As a matter of fact, > we have not heard of complaints of such performance regression issues in > batch mode. > > The primary goal of this FLIP is to allow the operator to run at the same > throughput (in stream mode when there is backlog) as it can currently do in > batch mode. And this goal is not affected by the disk overhead issue > mentioned above. > > I am thinking maybe we can treat it as an
Re:Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
Hello Jing, Thanks for your reply. The URI field should be final and the Path will be immutable.The static method deserializeFromDataInputView will create a new Path object instead of replacing the URI field in a existed Path Object. For the crossing multiple modules issue, I've explained it in the reply to Matthias. Best regards, Wencong Liu At 2023-07-21 18:05:26, "Jing Ge" wrote: >Hi Wencong, > >Just out of curiosity, will the newly introduced >deserializeFromDataInputView() method make the Path mutable again? > >What Matthias suggested makes sense, although the extension might make this >FLIP cross multiple modules. > >Best regards, >Jing > >On Fri, Jul 21, 2023 at 10:23 AM Matthias Pohl > wrote: > >> There's a kind-of-related issue FLINK-4758 [1] that proposes removing the >> IOReadableWritable interface from more classes. It was briefly mentioned in >> the must-have work items discussion [2]. >> >> I'm not too sure about the usage of IOReadableWritable: ...whether it would >> go away with the removal of the DataSet API in general (the Jira issue has >> DataSet as a component), anyway. >> >> Otherwise, might it make sense to extend the scope of this FLIP? >> >> [1] https://issues.apache.org/jira/browse/FLINK-4758 >> [2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r >> >> On Fri, Jul 21, 2023 at 6:04 AM Xintong Song >> wrote: >> >> > +1 >> > >> > Best, >> > >> > Xintong >> > >> > >> > >> > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu >> wrote: >> > >> > > Hi devs, >> > > >> > > I would like to start a discussion on FLIP-347: Remove >> IOReadableWritable >> > > serialization in Path [1]. >> > > >> > > >> > > The Path class is currently mutable to support IOReadableWritable >> > > serialization. However, many parts >> > > of the code assume that the Path is immutable. By making the Path class >> > > immutable, we can ensure >> > > that paths are stored correctly without the possibility of mutation and >> > > eliminate the occurrence of subtle errors. >> > > As such I propose to modify the Path class to no longer implement the >> > > IOReadableWritable interface. >> > > Looking forward to your feedback. >> > > [1] >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path >> > > Best regards, >> > > >> > > >> > > Wencong Liu >> > >>
Re:Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
Hello Matthias, Thanks for your response. As described in FLIP-347, I have thoroughly reviewed all instances where the IOReadableWritable serialization of the Path class is used, and I have identified three classes that still utilize it for de/serializing the Path. 1. FileSourceSplitSerializer: This class de/serializes the Path during the process of de/serializing the FileSourceSplit. 2. Two tests: TestManagedSinkCommittableSerializer, TestManagedFileSourceSplitSerializer. For 1, it can be modified to not serialize the Path by IOReadableWritable. For 2, these two tests will be deprecated before Flink 2.0. BTW, in my POC of this FLIP, I removed the IOReadableWritable usage in 1/2 and it has passed the CI on Azure. I seems that the IOReadableWritable of Path is not used by DataSet API or other components. I have documented these findings in FLINK-5336 [1]. Best regards, Wencong Liu [1] https://issues.apache.org/jira/browse/FLINK-5336 At 2023-07-21 16:22:34, "Matthias Pohl" wrote: >There's a kind-of-related issue FLINK-4758 [1] that proposes removing the >IOReadableWritable interface from more classes. It was briefly mentioned in >the must-have work items discussion [2]. > >I'm not too sure about the usage of IOReadableWritable: ...whether it would >go away with the removal of the DataSet API in general (the Jira issue has >DataSet as a component), anyway. > >Otherwise, might it make sense to extend the scope of this FLIP? > >[1] https://issues.apache.org/jira/browse/FLINK-4758 >[2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r > >On Fri, Jul 21, 2023 at 6:04 AM Xintong Song wrote: > >> +1 >> >> Best, >> >> Xintong >> >> >> >> On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu wrote: >> >> > Hi devs, >> > >> > I would like to start a discussion on FLIP-347: Remove IOReadableWritable >> > serialization in Path [1]. >> > >> > >> > The Path class is currently mutable to support IOReadableWritable >> > serialization. However, many parts >> > of the code assume that the Path is immutable. By making the Path class >> > immutable, we can ensure >> > that paths are stored correctly without the possibility of mutation and >> > eliminate the occurrence of subtle errors. >> > As such I propose to modify the Path class to no longer implement the >> > IOReadableWritable interface. >> > Looking forward to your feedback. >> > [1] >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path >> > Best regards, >> > >> > >> > Wencong Liu >>
Re: [DISUCSS] Deprecate multiple APIs in 1.18
got it, thanks for the clarification. Best regards, Jing On Fri, Jul 21, 2023 at 5:14 AM Jane Chan wrote: > Hi, Jing, > > I share the same opinion with Xintong. The base class `TimestampExtractor` > for these two classes was deprecated in release-1.12[1][2], so deprecating > these two subclasses is relatively straightforward. > > [1] https://issues.apache.org/jira/browse/FLINK-19453 > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces > > > Best, > Jane > > On Fri, Jul 21, 2023 at 10:19 AM Xintong Song > wrote: > > > I personally don't think it's necessary. IIUC, we have already decided to > > deprecate TableSource / TableSink, and not marking StreamRecordTimestamp > > and ExistingField deprecated is just an oversight in carrying out that > > decision. > > > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jul 21, 2023 at 7:48 AM Jing Ge > > wrote: > > > > > Hi Jane, Hi Xintong, > > > > > > The specific discussion of deprecating `StreamRecordTimestamp` & > > > `ExistingField` was kind of hiding in the discussion of the umbrella > > topic > > > that people might miss out. Does it make sense to start an > > > individual Discussion thread for it like we deprecate any other APIs? > > > > > > For all APIs that should be marked as @PublicEvolving, dedicated > > graduation > > > threads are recommended. > > > > > > Best regards, > > > Jing > > > > > > > > > On Wed, Jul 19, 2023 at 10:09 AM Xintong Song > > > wrote: > > > > > > > Thank you, Jane. > > > > > > > > What you said (preparing a FLIP for ManagedTable related classes, not > > > > deprecating SourceFunctionProvider, and starting a dedicated > discussion > > > for > > > > the missing annotations) sounds good to me. > > > > > > > > In addition, if there's no objections from the community on marking > > > > `StreamRecordTimestamp` & `ExistingField` as deprecated, you may > > consider > > > > creating a Jira ticket for it as a sub-task of FLINK-32557. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Wed, Jul 19, 2023 at 3:48 PM Jane Chan > > wrote: > > > > > > > > > Hi Xintong, > > > > > > > > > > > IIUC, you are suggesting to mark the classes with a red > background > > as > > > > > `@Deprecated` in 1.18? > > > > > > > > > > Exactly. They are not marked at the code level but suppose to be > > > > > deprecated. > > > > > > > > > > > - For the ManagedTable related classes, is there any FLIP > > explicitly > > > > that > > > > > decides to deprecate them? If not, I think it would be nice to have > > > one, > > > > to > > > > > formally decide the deprecation with a vote. I'd expect such a FLIP > > can > > > > be > > > > > quite lightweight. > > > > > > > > > > Currently, there's no formal FLIP yet. I'd like to prepare one to > > > > initiate > > > > > the deprecation process. > > > > > > > > > > > - For `SourceFunctionProvider`, please beware there are recently > > some > > > > > objections against deprecating `SourceFunction`, and the code > changes > > > > > marking it as `@Deprecated` might be reverted. See [1][2] for more > > > > details. > > > > > So my question is, if `SourceFunction` will not be deprecated until > > all > > > > > sub-tasks in FLINK-28045 are resolved, do you still think we should > > > > > deprecate `SourceFunctionProvider` now? > > > > > > > > > > Thanks for the reminder. If `SourceFunction` is no longer being > > > > deprecated, > > > > > then `SourceFunctionProvider` also needs to be retained. I have > > updated > > > > the > > > > > sheet and removed `SourceFunctionProvider`. > > > > > > > > > > > I'm in general +1 to add the missing API annotations. However, I > > > don't > > > > > have the expertise to comment on the classes and suggestd API > levels > > > > being > > > > > listed. > > > > > > > > > > I've updated the sheet and added comments on all the APIs that are > > > > > suggested to be marked as PublicEvolving, explaining the reasons. > The > > > > rest > > > > > APIs are either util classes or implementations and hence are > > suggested > > > > to > > > > > be Internal. I can start a discussion about the suggested API level > > to > > > > find > > > > > developers who can help review them. > > > > > > > > > > Best, > > > > > Jane > > > > > > > > > > On Wed, Jul 19, 2023 at 12:22 PM Xintong Song < > tonysong...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Thanks for the beautiful sheets, Jane. > > > > > > > > > > > > 1. This sheet < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1dZBNHLuAHYJt3pFU8ZtfUzrYyf2ZFQ6wybDXGS1bHno/edit?usp=sharing > > > > > > > > > > > > > > summarizes the user-facing classes and methods that need to be > > > > > deprecated > > > > > > > under the flink-table module, some of which are marked with a > red > > > > > > > background and belong to the APIs that need to be depreciated > but > > > are > > > > > not > > > > > > > explicitly marked in
[jira] [Created] (FLINK-32641) json format supports pojo type
Jacky Lau created FLINK-32641: - Summary: json format supports pojo type Key: FLINK-32641 URL: https://issues.apache.org/jira/browse/FLINK-32641 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.18.0 Reporter: Jacky Lau Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Status of Statefun Project
This is great news, as we're using statefun as well. Please don't hesitate to let me know if you need me to do some additional testing on a real life prod-like setup. On Sat, 24 Jun 2023 at 18:41, Galen Warren via user wrote: > Great -- thanks! > > I'm going to be out of town for about a week but I'll take a look at this > when I'm back. > > On Tue, Jun 20, 2023 at 8:46 AM Martijn Visser > wrote: > >> Hi Galen, >> >> Yes, I'll be more than happy to help with Statefun releases. >> >> Best regards, >> >> Martijn >> >> On Tue, Jun 20, 2023 at 2:21 PM Galen Warren >> wrote: >> >>> Thanks. >>> >>> Martijn, to answer your question, I'd need to do a small amount of work >>> to get a PR ready, but not much. Happy to do it if we're deciding to >>> restart Statefun releases -- are we? >>> >>> -- Galen >>> >>> On Sat, Jun 17, 2023 at 9:47 AM Tzu-Li (Gordon) Tai >>> wrote: >>> > Perhaps he could weigh in on whether the combination of automated tests plus those smoke tests should be sufficient for testing with new Flink versions What we usually did at the bare minimum for new StateFun releases was the following: 1. Build tests (including the smoke tests in the e2e module, which covers important tests like exactly-once verification) 2. Updating the flink-statefun-playground repo and manually running all language examples there. If upgrading Flink versions was the only change in the release, I'd probably say that this is sufficient. Best, Gordon On Thu, Jun 15, 2023 at 5:25 AM Martijn Visser < martijnvis...@apache.org> wrote: > Let me know if you have a PR for a Flink update :) > > On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user < > u...@flink.apache.org> wrote: > >> Thanks Martijn. >> >> Personally, I'm already using a local fork of Statefun that is >> compatible with Flink 1.16.x, so I wouldn't have any need for a released >> version compatible with 1.15.x. I'd be happy to do the PRs to modify >> Statefun to work with new versions of Flink as they come along. >> >> As for testing, Statefun does have unit tests and Gordon also sent me >> instructions a while back for how to do some additional smoke tests which >> are pretty straightforward. Perhaps he could weigh in on whether the >> combination of automated tests plus those smoke tests should be >> sufficient >> for testing with new Flink versions (I believe the answer is yes). >> >> -- Galen >> >> >> >> On Thu, Jun 8, 2023 at 8:01 AM Martijn Visser < >> martijnvis...@apache.org> wrote: >> >>> Hi all, >>> >>> Apologies for the late reply. >>> >>> I'm willing to help out with merging requests in Statefun to keep >>> them >>> compatible with new Flink releases and create new releases. I do >>> think that >>> validation of the functionality of these releases depends a lot on >>> those >>> who do these compatibility updates, with PMC members helping out >>> with the >>> formal process. >>> >>> > Why can't the Apache Software Foundation allow community members >>> to bring >>> it up to date? >>> >>> There's nothing preventing anyone from reviewing any of the current >>> PRs or >>> opening new ones. However, none of them are approved [1], so there's >>> also >>> nothing to merge. >>> >>> > I believe that there are people and companies on this mailing list >>> interested in supporting Apache Flink Stateful Functions. >>> >>> If so, then now is the time to show. >>> >>> Would there be a preference to create a release with Galen's merged >>> compatibility update to Flink 1.15.2, or do we want to skip that and >>> go >>> straight to a newer version? >>> >>> Best regards, >>> >>> Martijn >>> >>> [1] >>> >>> https://github.com/apache/flink-statefun/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved >>> >>> On Tue, Jun 6, 2023 at 3:55 PM Marco Villalobos < >>> mvillalo...@kineteque.com> >>> wrote: >>> >>> > Why can't the Apache Software Foundation allow community members >>> to bring >>> > it up to date? >>> > >>> > What's the process for that? >>> > >>> > I believe that there are people and companies on this mailing list >>> > interested in supporting Apache Flink Stateful Functions. >>> > >>> > You already had two people on this thread express interest. >>> > >>> > At the very least, we could keep the library versions up to date. >>> > >>> > There are only a small list of new features that might be >>> worthwhile: >>> > >>> > 1. event time processing >>> > 2. state rest api >>> > >>> > >>> > On Jun 6, 2023, at 3:06 AM, Chesnay Schepler >>> wrote: >>> > >>> > If you were to fork it
Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open
+1 On Fri, Jul 21, 2023 at 10:22 AM Yuxin Tan wrote: > +1 > > Best, > Yuxin > > > Xintong Song 于2023年7月21日周五 12:04写道： > > > +1 > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu > wrote: > > > > > Hi devs, > > > > > > I would like to start a discussion on FLIP-344: Remove parameter in > > > RichFunction#open [1]. > > > > > > The open() method in RichFunction requires a Configuration instance as > an > > > argument, > > > which is always passed as a new instance without any configuration > > > parameters in > > > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include this > > > parameter > > > in the open() method. > > > As such I propose to remove the Configuration field from > > > RichFunction#open(Configuration parameters). > > > Looking forward to your feedback. > > > [1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231 > > > Best regards, > > > > > > > > > Wencong Liu > > >
Re: [VOTE] Release 2.0 must-have work items
On 21/07/2023 11:45, Leonard Xu wrote: In this way, the user will see the deprecated API firstly but they can not find a candidate if we can not finish all tasks in one minor version . i'm not convinced that this matters. There will be a whole bunch of APIs deprecated in 1.18 (that will remain in 1.x!) without a replacement so we can remove them in 2.0. We already accepted this scenario.
Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
Hi Wencong, Just out of curiosity, will the newly introduced deserializeFromDataInputView() method make the Path mutable again? What Matthias suggested makes sense, although the extension might make this FLIP cross multiple modules. Best regards, Jing On Fri, Jul 21, 2023 at 10:23 AM Matthias Pohl wrote: > There's a kind-of-related issue FLINK-4758 [1] that proposes removing the > IOReadableWritable interface from more classes. It was briefly mentioned in > the must-have work items discussion [2]. > > I'm not too sure about the usage of IOReadableWritable: ...whether it would > go away with the removal of the DataSet API in general (the Jira issue has > DataSet as a component), anyway. > > Otherwise, might it make sense to extend the scope of this FLIP? > > [1] https://issues.apache.org/jira/browse/FLINK-4758 > [2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r > > On Fri, Jul 21, 2023 at 6:04 AM Xintong Song > wrote: > > > +1 > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu > wrote: > > > > > Hi devs, > > > > > > I would like to start a discussion on FLIP-347: Remove > IOReadableWritable > > > serialization in Path [1]. > > > > > > > > > The Path class is currently mutable to support IOReadableWritable > > > serialization. However, many parts > > > of the code assume that the Path is immutable. By making the Path class > > > immutable, we can ensure > > > that paths are stored correctly without the possibility of mutation and > > > eliminate the occurrence of subtle errors. > > > As such I propose to modify the Path class to no longer implement the > > > IOReadableWritable interface. > > > Looking forward to your feedback. > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path > > > Best regards, > > > > > > > > > Wencong Liu > > >
Re: [VOTE] Release 2.0 must-have work items
> @Leonard > If we follow the agreed-upon and voted path and do not revert [1], all the > formalities get fulfilled Hi Alexander Sorry for making you uncomfortable even though it was discussed on the dev list and under JIRA before reverting this PR. I'm also +1 for the FLIP as your posted. The reason to revert the PR is not -1 for the voted FLIP, the reason we revert the PR is the incorrect subtask implementation order. We should not mark the API as deprecated firstly and then to implement the necessary subtasks. In this way, the user will see the deprecated API firstly but they can not find a candidate if we can not finish all tasks in one minor version . In this case, there're 7 subtasks to resolve and only 3 days left for 1.18 feature freeze, that means we have a high probability of releasing a deprecated API but not providing a migration path. It looks like that we voted a FLIP for a feature Flink supports XXX, we first add this feature to the document to tell users that Flink supports t XXX, but this feature may needs many subtasks to be completed in one or more versions. So, before we actually support XXX, it is incorrect for users to see this document that Flink supports XXX, we should add the document after we have finished all subtasks of the voted feature. Please correct me if I wrongly understand the order of sub-task. Best, Leonard > >> Hi Alexander >> >>> I see your concerns regarding the complexity of migration, but we still >>> have one year to address them. >> >> Not only the complexity of migration, but also we lack migration path for >> now. We have to deprecate SourceFunction/SinkFunction in 1.18 which feature >> freeze date is 2023/07/24 CEST If we want to remove >> SourceFunction/SinkFunction in 2.0, remove a public API requires at least >> two minor versions. There’re many subtasks to finish before we can mark >> them as deprecated[1], I think we have no time to finish these subtasks >> this week(hint: not this year). That’s why we suggested removing the >> SourceFunction/SinkFunction item from must to have. >> >> Best, >> Leonard >> [1]https://issues.apache.org/jira/browse/FLINK-28045 >> >>
Re: [DISCUSS] Feature freeze and deprecation work for 2.0
Good question. CC-ed the release managers. My 2-cents: I think the purpose of feature freeze is to prevent new feature / improvement changes from destabilizing the code base, in order to get a stable and verified release. Based on this, I'd suggest: - Considering FLIPs that purely mark an API as deprecated and do not introduce anything new as "not a new feature", because that can hardly cause any trouble. - Considering FLIPs that introduce new / replacing APIs in addition to deprecating old ones as "new features or improvements". Merging codes for such FLIPs after the feature freeze should be carefully evaluated and require permissions from the release managers. - Further extending the feature freeze might also be an option, but I personally don't think it's necessary to block the release testing. Most of the recent API deprecation FLIPs require only minor changes that IHMO can barely affect the stability of the codebase. But this should be the release managers' call. Looking forward to your opinions. Best, Xintong On Fri, Jul 21, 2023 at 4:25 PM Matthias Pohl wrote: > The feature freeze was postponed to July 24 (end of this week in > Europe/early morning Monday in East Asia) in [1]. What's the 1.18 release > managers' take on all the FLIPs that were recently started and require some > deprecation work (which ideally should go into 1.18)? How does that work > with the feature freeze happening by the end of this week? > > - Do we plan to extend the feature freeze to allow deprecation changes to > go in? > - Do we consider depreciation work to be "not a new feature" which means > that we're ok with merging them after the feature freeze? > > Best, > Matthias > > [1] https://lists.apache.org/thread/9fck1m5llrnv5gx1od05tzx84oy0b4z0 >
Re: FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables
There's a separate discussion altogether. On 21/07/2023 02:49, Mason Chen wrote: Hi all, I agree getScopeVariables is in line with the existing terminology but I’ve seen that the existing terminology is a bit confusing with regards to how users end up querying these metrics and building alerts/dashboards. I often get the question how do I add a tag or label to my Flink metric, and it would be more intuitive if labels and tags replaced group as a name, to be more in align with popular metric systems. Best, Mason On Thu, Jul 20, 2023 at 1:41 AM Chesnay Schepler wrote: environment variables is a very different though though, that'd just confuse users. It's also not a term we've used in the documentation. getScopeVariables would I guess be most in line with existing terminology. On 19/07/2023 18:10, Matthias Pohl wrote: We don't have a well-defined process for breaking behavioral changes. We could consider adding a new method with a different name. Introducing a new API to make the behavioral change visible was also the suggestion in the deprecation ML thread [1]. getEnvironmentVariables (or even getEnvironment) might be a reasonable change. [1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 On Tue, Jul 18, 2023 at 1:10 PM Jing Ge wrote: +1 On Tue, Jul 18, 2023 at 12:24 PM Xintong Song wrote: +1 Best, Xintong On Tue, Jul 18, 2023 at 5:02 PM Chesnay Schepler wrote: The FLIP number was changed to 342. On 18/07/2023 10:56, Chesnay Schepler wrote: MetricGroup#getAllVariables returns all variables associated with the metric, for example: | = abcde| | = ||0| The keys are surrounded by brackets for no particular reason. In virtually every use-case for this method the user is stripping the brackets from keys, as done in: * our datadog reporter: https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java#L244 < https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java#L244 * our prometheus reporter (implicitly via a character filter): https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L236 < https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L236 * our JMX reporter: https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-jmx/src/main/java/org/apache/flink/metrics/jmx/JMXReporter.java#L223 < https://github.com/apache/flink/blob/9c3c8afbd9325b5df8291bd831da2d9f8785b30a/flink-metrics/flink-metrics-jmx/src/main/java/org/apache/flink/metrics/jmx/JMXReporter.java#L223 I propose to change the method spec and implementation to remove the brackets around keys. For migration purposes it may make sense to add a new method with the new behavior (|getVariables()|) and deprecate the old method. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425202
[DISCUSS] Feature freeze and deprecation work for 2.0
The feature freeze was postponed to July 24 (end of this week in Europe/early morning Monday in East Asia) in [1]. What's the 1.18 release managers' take on all the FLIPs that were recently started and require some deprecation work (which ideally should go into 1.18)? How does that work with the feature freeze happening by the end of this week? - Do we plan to extend the feature freeze to allow deprecation changes to go in? - Do we consider depreciation work to be "not a new feature" which means that we're ok with merging them after the feature freeze? Best, Matthias [1] https://lists.apache.org/thread/9fck1m5llrnv5gx1od05tzx84oy0b4z0
Re: [VOTE] Release 2.0 must-have work items
> How about this, we continue with the vote as is, and keep the discussion on the SourceFunction in Jira or a separate thread. Sure, but I just want to mention two important things here before we switch over to [1]: >Given that eliminating the removal of SourceFunction was proposed 10 days - This is not the case - here is the original discussion thread from a year ago [2] - Here are the vote results [3] agreeing to do exactly the following: > This proposition implies marking the SourceFunction interface > itself as @Deprecated + redirecting to the FLIP-27 Source API > right away, without waiting for all the subtasks to be completed. This was all already discussed and agreed upon, the decision to revert the change comes a bit as a surprise. @Leonard If we follow the agreed-upon and voted path and do not revert [1], all the formalities get fulfilled. The rest of the discussion can happen in [1] [1] https://issues.apache.org/jira/browse/FLINK-28046 [2] https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 [3] https://lists.apache.org/thread/hrpsddgz65hjvhjozhg72s0wsmxz145p Thanks, Alex On Fri, 21 Jul 2023 at 04:49, Leonard Xu wrote: > Hi Alexander > > > I see your concerns regarding the complexity of migration, but we still > > have one year to address them. > > Not only the complexity of migration, but also we lack migration path for > now. We have to deprecate SourceFunction/SinkFunction in 1.18 which feature > freeze date is 2023/07/24 CEST If we want to remove > SourceFunction/SinkFunction in 2.0, remove a public API requires at least > two minor versions. There’re many subtasks to finish before we can mark > them as deprecated[1], I think we have no time to finish these subtasks > this week(hint: not this year). That’s why we suggested removing the > SourceFunction/SinkFunction item from must to have. > > Best, > Leonard > [1]https://issues.apache.org/jira/browse/FLINK-28045 > >
Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
There's a kind-of-related issue FLINK-4758 [1] that proposes removing the IOReadableWritable interface from more classes. It was briefly mentioned in the must-have work items discussion [2]. I'm not too sure about the usage of IOReadableWritable: ...whether it would go away with the removal of the DataSet API in general (the Jira issue has DataSet as a component), anyway. Otherwise, might it make sense to extend the scope of this FLIP? [1] https://issues.apache.org/jira/browse/FLINK-4758 [2] https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r On Fri, Jul 21, 2023 at 6:04 AM Xintong Song wrote: > +1 > > Best, > > Xintong > > > > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu wrote: > > > Hi devs, > > > > I would like to start a discussion on FLIP-347: Remove IOReadableWritable > > serialization in Path [1]. > > > > > > The Path class is currently mutable to support IOReadableWritable > > serialization. However, many parts > > of the code assume that the Path is immutable. By making the Path class > > immutable, we can ensure > > that paths are stored correctly without the possibility of mutation and > > eliminate the occurrence of subtle errors. > > As such I propose to modify the Path class to no longer implement the > > IOReadableWritable interface. > > Looking forward to your feedback. > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path > > Best regards, > > > > > > Wencong Liu >
Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open
+1 Best, Yuxin Xintong Song 于2023年7月21日周五 12:04写道： > +1 > > Best, > > Xintong > > > > On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu wrote: > > > Hi devs, > > > > I would like to start a discussion on FLIP-344: Remove parameter in > > RichFunction#open [1]. > > > > The open() method in RichFunction requires a Configuration instance as an > > argument, > > which is always passed as a new instance without any configuration > > parameters in > > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include this > > parameter > > in the open() method. > > As such I propose to remove the Configuration field from > > RichFunction#open(Configuration parameters). > > Looking forward to your feedback. > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231 > > Best regards, > > > > > > Wencong Liu >
Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path
+1 Best, Yuxin Xintong Song 于2023年7月21日周五 12:04写道： > +1 > > Best, > > Xintong > > > > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu wrote: > > > Hi devs, > > > > I would like to start a discussion on FLIP-347: Remove IOReadableWritable > > serialization in Path [1]. > > > > > > The Path class is currently mutable to support IOReadableWritable > > serialization. However, many parts > > of the code assume that the Path is immutable. By making the Path class > > immutable, we can ensure > > that paths are stored correctly without the possibility of mutation and > > eliminate the occurrence of subtle errors. > > As such I propose to modify the Path class to no longer implement the > > IOReadableWritable interface. > > Looking forward to your feedback. > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path > > Best regards, > > > > > > Wencong Liu >
Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()
+1 Best, Yuxin Jing Ge 于2023年7月21日周五 15:41写道： > +1 > > NIT: the release in the FLIP is still empty, it should be 2.0 > > Best regards, > Jing > > On Fri, Jul 21, 2023 at 6:03 AM Xintong Song > wrote: > > > +1 > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu > wrote: > > > > > Hi devs, > > > > > > I would like to start a discussion on FLIP-343: Remove parameter in > > > WindowAssigner#getDefaultTrigger() [1]. > > > > > > > > > The method getDefaultTrigger() in WindowAssigner takes a > > > StreamExecutionEnvironment > > > parameter, but this parameter is not actually used for any subclasses > of > > > WindowAssigner. > > > Therefore, it is unnecessary to include this parameter. > > > As such I propose to remove the StreamExecutionEnvironment field from > > > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env). > > > Looking forward to your feedback. > > > [1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229 > > > Best regards, > > > > > > > > > Wencong Liu > > >
Re: [VOTE] FLIP-346: Deprecate ManagedTable related APIs
+1 Best, Lincoln Lee Leonard Xu 于2023年7月21日周五 16:07写道： > +1 > > Best, > Leonard > > > On Jul 21, 2023, at 4:02 PM, yuxia wrote: > > > > +1（binging） > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Jane Chan" > > 收件人: "dev" > > 发送时间: 星期五, 2023年 7 月 21日 下午 3:41:11 > > 主题: [VOTE] FLIP-346: Deprecate ManagedTable related APIs > > > > Hi developers, > > > > Thanks for all the feedback on FLIP-346: Deprecate ManagedTable related > > APIs[1]. > > Based on the discussion[2], we have reached a consensus, so I would like > to > > start a vote. > > > > The vote will last for at least 72 hours unless there is an objection or > > insufficient votes. > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs > > [2] https://lists.apache.org/thread/5dvqyhqp5fbtm54944xohts71modwd99 > > > > Best, > > Jane > >
Re: [VOTE] FLIP-346: Deprecate ManagedTable related APIs
+1 Best, Leonard > On Jul 21, 2023, at 4:02 PM, yuxia wrote: > > +1（binging） > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Jane Chan" > 收件人: "dev" > 发送时间: 星期五, 2023年 7 月 21日 下午 3:41:11 > 主题: [VOTE] FLIP-346: Deprecate ManagedTable related APIs > > Hi developers, > > Thanks for all the feedback on FLIP-346: Deprecate ManagedTable related > APIs[1]. > Based on the discussion[2], we have reached a consensus, so I would like to > start a vote. > > The vote will last for at least 72 hours unless there is an objection or > insufficient votes. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs > [2] https://lists.apache.org/thread/5dvqyhqp5fbtm54944xohts71modwd99 > > Best, > Jane
Re: [VOTE] FLIP-346: Deprecate ManagedTable related APIs
+1（binging） Best regards, Yuxia - 原始邮件 - 发件人: "Jane Chan" 收件人: "dev" 发送时间: 星期五, 2023年 7 月 21日 下午 3:41:11 主题: [VOTE] FLIP-346: Deprecate ManagedTable related APIs Hi developers, Thanks for all the feedback on FLIP-346: Deprecate ManagedTable related APIs[1]. Based on the discussion[2], we have reached a consensus, so I would like to start a vote. The vote will last for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs [2] https://lists.apache.org/thread/5dvqyhqp5fbtm54944xohts71modwd99 Best, Jane
[VOTE] FLIP-346: Deprecate ManagedTable related APIs
Hi developers, Thanks for all the feedback on FLIP-346: Deprecate ManagedTable related APIs[1]. Based on the discussion[2], we have reached a consensus, so I would like to start a vote. The vote will last for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs [2] https://lists.apache.org/thread/5dvqyhqp5fbtm54944xohts71modwd99 Best, Jane
Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()
+1 NIT: the release in the FLIP is still empty, it should be 2.0 Best regards, Jing On Fri, Jul 21, 2023 at 6:03 AM Xintong Song wrote: > +1 > > Best, > > Xintong > > > > On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu wrote: > > > Hi devs, > > > > I would like to start a discussion on FLIP-343: Remove parameter in > > WindowAssigner#getDefaultTrigger() [1]. > > > > > > The method getDefaultTrigger() in WindowAssigner takes a > > StreamExecutionEnvironment > > parameter, but this parameter is not actually used for any subclasses of > > WindowAssigner. > > Therefore, it is unnecessary to include this parameter. > > As such I propose to remove the StreamExecutionEnvironment field from > > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env). > > Looking forward to your feedback. > > [1] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229 > > Best regards, > > > > > > Wencong Liu >
Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs
Hi, all, Since there are no objections against the FLIP, I'm preparing to start a vote. Please don't hesitate to follow up on this if you have any concerns. Best, Jane On Fri, Jul 21, 2023 at 10:19 AM Jane Chan wrote: > Hi Leonard, > > I got your point and adjusted the FLIP to cover this. Thanks so much. > > Best, > Jane > > On Fri, Jul 21, 2023 at 10:01 AM Leonard Xu wrote: > >> >> Hi, Jane >> >> I mean these pages, >> https://nightlies.apache.org/flink/flink-table-store-docs-stable/ < >> https://nightlies.apache.org/flink/flink-table-store-docs-stable/> >> >> Best, >> Leonard >> >> > On Jul 20, 2023, at 8:03 PM, Jane Chan wrote: >> > >> > Hi, all, >> > >> > Thanks for your opinions. >> > >> > @Leonard >> >> We can remove related document as well, Keeping a external link to >> Apache >> > Paimon should be enough. >> > >> > Thanks Leonard, for the kind reminder. I checked the JIRA[1] related to >> > FLIP-188, and there is no issue related to documentation. Meanwhile, the >> > documentation for managed tables has already been removed from Paimon's >> > documentation[2] earlier. >> > >> > Considering there are no objections from the community and Paimon PMC >> > @Jingsong has confirmed that deprecation has no impact on Paimon, I am >> > planning to initiate a vote tomorrow. >> > >> > If you have any questions, please feel free to follow up on this >> discussion. >> > >> > [1] https://issues.apache.org/jira/browse/FLINK-25152 >> > [2] https://paimon.apache.org/docs/master/how-to/creating-tables/ >> > >> > Best, >> > Jane >> > >> > >> > On Thu, Jul 20, 2023 at 4:58 PM Jing Ge >> wrote: >> > >> >> +1 >> >> >> >> On Thu, Jul 20, 2023 at 10:37 AM Lincoln Lee >> >> wrote: >> >> >> >>> +1 >> >>> >> >>> >> >>> Best, >> >>> Lincoln Lee >> >>> >> >>> >> >>> Leonard Xu 于2023年7月20日周四 16:26写道： >> >>> >> Thanks Jane for driving this FLIP. >> >> >> +1, >> >> Minor: We can remove related document as well, Keeping a external >> link >> >> to >> Apache Paimon should be enough. >> >> Best, >> Leonard >> >> >> >> > On Jul 20, 2023, at 3:07 PM, Martijn Visser < >> >> martijnvis...@apache.org> >> wrote: >> > >> > +1 >> > >> > On Thu, Jul 20, 2023 at 7:53 AM Xintong Song > > >> wrote: >> > >> >> +1 >> >> >> >> Best, >> >> >> >> Xintong >> >> >> >> >> >> >> >> On Thu, Jul 20, 2023 at 1:25 PM Jingsong Li < >> jingsongl...@gmail.com >> >>> >> >> wrote: >> >> >> >>> +1 >> >>> >> >>> On Thu, Jul 20, 2023 at 12:31 PM Jane Chan > > >> >> wrote: >> >> Hi, devs, >> >> I would like to start a discussion on FLIP-346: Deprecate >> >>> ManagedTable >> related APIs[1]. >> >> These APIs were initially designed for Flink Table Store, which >> >> has >> joined the Apache Incubator as a separate project called Apache >> Paimon(incubating). >> >> Since they are obsolete and not used by Paimon anymore, I propose >> >> to >> deprecate them in v1.18 and further remove them before v2.0. >> >> Looking forward to your feedback. >> >> [1] >> >> >>> >> >> >> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-346%3A+Deprecate+ManagedTable+related+APIs >> >> Best regards, >> Jane >> >>> >> >> >> >> >> >>> >> >> >> >>
