Re: Flink netty connector for TCP source
Yes, you are right ! On Thu, Aug 3, 2023 at 1:35 PM Kamal Mittal wrote: > Hello Shammon, > > > > As it is said one split enumerator for one source means multiple sub-tasks > of that source (if parallelism >1) will use same split enumerator instance > right? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 03 August 2023 10:54 AM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > I think it depends on your practical application. In general, the built-in > source in Flink such as kafka or hive will proactively fetch splits from > source, instead of starting a service with a source service pushed over. > > > > Returning to the issue of port conflicts, you may need to check if the > specified port in the split enumerator is being used by other services, or > if random ports can be used. Generally speaking, there will only be one > split enumerator for one source, but when the job master fails to restart, > the split enumerator of the previous job may not be completely shutdown. If > they use the same port, it may also cause conflicts. > > > > Best, > > Shammon FY > > > > On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal > wrote: > > Hello Shammon, > > > > Please have a look for below and share views. > > > > Rgds, > > Kamal > > > > *From:* Kamal Mittal via user > *Sent:* 02 August 2023 08:02 AM > *To:* Shammon FY ; user@flink.apache.org > *Subject:* RE: Flink netty connector for TCP source > > > > Thanks Shammon. > > > > Purpose of opening server socket in Split Enumerator was that it has only > one instance per source and so the server socket too (port binding can > happen only once). And then accepted Socket connections > (serversocket.accept()) will act as splits which will be further processed > by readers. > > > > Let me know please if there is issue you see in above understanding or > some other way you can suggest. Issue is that server socket can only bind a > port once over a machine node and that’s why we thought to open it in split > enumerator which is called only once and per source. > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 02 August 2023 07:48 AM > *To:* Kamal Mittal ; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > It confuses me a little that what's the purpose of opening a server socket > in SplitEnumerator? Currently there will be only one SplitEnumerator > instance in JobManager for each one source not each source subtask. If > there's only one source in your job, no matter how much parallelism this > source has, there will be only one SplitEnumerator instance in JM which can > connect and send events to the source subtasks. > > > > Additionally, if you want to read data from a netty source, you can > implement your NettySplitEnumerator to get splits from netty source and > assign them to a NettySourceReader which will read data from netty source > according to the splits. > > > > Best, > > Shammon FY > > > > > > On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > I created a custom server socket source in which opened server socket in > split enumerator, source parallelism is =2 and it gives error while running > - “Address is already in use” i.e. it tried to open two server sockets as > per parallelism which is not correct as parallelism is applicable for > source operator and not for split enumerator? > > > > Please correct me if above understanding is not correct. > > > > Rgds, > > Kamal > > > > *From:* Hang Ruan > *Sent:* 01 August 2023 08:55 AM > *To:* Kamal Mittal > *Cc:* liu ron ; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal. > > > > The SplitEnumerator is contained in the SourceCoordinator. They are only > used in JM. > > > > Best, > > Hang > > > > Kamal Mittal via user 于2023年8月1日周二 10:43写道： > > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job Manager rather than at > Task manager? > > > > *From:* liu ron > *Sent:* 31 July 2023 10:06 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal > > Currently, we don't provide the connector like TCP source in Flink main
RE: Flink netty connector for TCP source
Hello Shammon, As it is said one split enumerator for one source means multiple sub-tasks of that source (if parallelism >1) will use same split enumerator instance right? Rgds, Kamal From: Shammon FY Sent: 03 August 2023 10:54 AM To: Kamal Mittal Cc: user@flink.apache.org Subject: Re: Flink netty connector for TCP source Hi Kamal, I think it depends on your practical application. In general, the built-in source in Flink such as kafka or hive will proactively fetch splits from source, instead of starting a service with a source service pushed over. Returning to the issue of port conflicts, you may need to check if the specified port in the split enumerator is being used by other services, or if random ports can be used. Generally speaking, there will only be one split enumerator for one source, but when the job master fails to restart, the split enumerator of the previous job may not be completely shutdown. If they use the same port, it may also cause conflicts. Best, Shammon FY On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Please have a look for below and share views. Rgds, Kamal From: Kamal Mittal via user mailto:user@flink.apache.org>> Sent: 02 August 2023 08:02 AM To: Shammon FY mailto:zjur...@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: RE: Flink netty connector for TCP source Thanks Shammon. Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers. Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source. Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 02 August 2023 07:48 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator? Please correct me if above understanding is not correct. Rgds, Kamal From: Hang Ruan mailto:ruanhang1...@gmail.com>> Sent: 01 August 2023 08:55 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: liu ron mailto:ron9@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron mailto:ron9@gmail.com>> Sent: 31 July 2023 10:06 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, J
Re: Flink netty connector for TCP source
Hi Kamal, I think it depends on your practical application. In general, the built-in source in Flink such as kafka or hive will proactively fetch splits from source, instead of starting a service with a source service pushed over. Returning to the issue of port conflicts, you may need to check if the specified port in the split enumerator is being used by other services, or if random ports can be used. Generally speaking, there will only be one split enumerator for one source, but when the job master fails to restart, the split enumerator of the previous job may not be completely shutdown. If they use the same port, it may also cause conflicts. Best, Shammon FY On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal wrote: > Hello Shammon, > > > > Please have a look for below and share views. > > > > Rgds, > > Kamal > > > > *From:* Kamal Mittal via user > *Sent:* 02 August 2023 08:02 AM > *To:* Shammon FY ; user@flink.apache.org > *Subject:* RE: Flink netty connector for TCP source > > > > Thanks Shammon. > > > > Purpose of opening server socket in Split Enumerator was that it has only > one instance per source and so the server socket too (port binding can > happen only once). And then accepted Socket connections > (serversocket.accept()) will act as splits which will be further processed > by readers. > > > > Let me know please if there is issue you see in above understanding or > some other way you can suggest. Issue is that server socket can only bind a > port once over a machine node and that’s why we thought to open it in split > enumerator which is called only once and per source. > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 02 August 2023 07:48 AM > *To:* Kamal Mittal ; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > It confuses me a little that what's the purpose of opening a server socket > in SplitEnumerator? Currently there will be only one SplitEnumerator > instance in JobManager for each one source not each source subtask. If > there's only one source in your job, no matter how much parallelism this > source has, there will be only one SplitEnumerator instance in JM which can > connect and send events to the source subtasks. > > > > Additionally, if you want to read data from a netty source, you can > implement your NettySplitEnumerator to get splits from netty source and > assign them to a NettySourceReader which will read data from netty source > according to the splits. > > > > Best, > > Shammon FY > > > > > > On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > I created a custom server socket source in which opened server socket in > split enumerator, source parallelism is =2 and it gives error while running > - “Address is already in use” i.e. it tried to open two server sockets as > per parallelism which is not correct as parallelism is applicable for > source operator and not for split enumerator? > > > > Please correct me if above understanding is not correct. > > > > Rgds, > > Kamal > > > > *From:* Hang Ruan > *Sent:* 01 August 2023 08:55 AM > *To:* Kamal Mittal > *Cc:* liu ron ; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal. > > > > The SplitEnumerator is contained in the SourceCoordinator. They are only > used in JM. > > > > Best, > > Hang > > > > Kamal Mittal via user 于2023年8月1日周二 10:43写道： > > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job Manager rather than at > Task manager? > > > > *From:* liu ron > *Sent:* 31 July 2023 10:06 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal > > Currently, we don't provide the connector like TCP source in Flink main > repo. If you need this connector, you can try to implement it refer to the > FLIP-27 source docs > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ > > > > Best, > > Ron > > > > Shammon FY 于2023年7月27日周四 11:23写道： > > I cannot find any information about netty source in flink website and it > is not in the connector list[1], so I'm think that it is not supported by > flink community > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ > > > > Best, > > Shammon FY > > &g
RE: Flink netty connector for TCP source
Hello Shammon, Please have a look for below and share views. Rgds, Kamal From: Kamal Mittal via user Sent: 02 August 2023 08:02 AM To: Shammon FY ; user@flink.apache.org Subject: RE: Flink netty connector for TCP source Thanks Shammon. Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers. Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source. Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 02 August 2023 07:48 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator? Please correct me if above understanding is not correct. Rgds, Kamal From: Hang Ruan mailto:ruanhang1...@gmail.com>> Sent: 01 August 2023 08:55 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: liu ron mailto:ron9@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron mailto:ron9@gmail.com>> Sent: 31 July 2023 10:06 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink? Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 27 July 2023 08:15 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
RE: Flink netty connector for TCP source
Thanks Shammon. Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers. Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source. Rgds, Kamal From: Shammon FY Sent: 02 August 2023 07:48 AM To: Kamal Mittal ; user@flink.apache.org Subject: Re: Flink netty connector for TCP source Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator? Please correct me if above understanding is not correct. Rgds, Kamal From: Hang Ruan mailto:ruanhang1...@gmail.com>> Sent: 01 August 2023 08:55 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: liu ron mailto:ron9@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron mailto:ron9@gmail.com>> Sent: 31 July 2023 10:06 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink? Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 27 July 2023 08:15 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
Re: Flink netty connector for TCP source
Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user wrote: > Hello, > > > > I created a custom server socket source in which opened server socket in > split enumerator, source parallelism is =2 and it gives error while running > - “Address is already in use” i.e. it tried to open two server sockets as > per parallelism which is not correct as parallelism is applicable for > source operator and not for split enumerator? > > > > Please correct me if above understanding is not correct. > > > > Rgds, > > Kamal > > > > *From:* Hang Ruan > *Sent:* 01 August 2023 08:55 AM > *To:* Kamal Mittal > *Cc:* liu ron ; user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal. > > > > The SplitEnumerator is contained in the SourceCoordinator. They are only > used in JM. > > > > Best, > > Hang > > > > Kamal Mittal via user 于2023年8月1日周二 10:43写道： > > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job Manager rather than at > Task manager? > > > > *From:* liu ron > *Sent:* 31 July 2023 10:06 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal > > Currently, we don't provide the connector like TCP source in Flink main > repo. If you need this connector, you can try to implement it refer to the > FLIP-27 source docs > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ > > > > Best, > > Ron > > > > Shammon FY 于2023年7月27日周四 11:23写道： > > I cannot find any information about netty source in flink website and it > is not in the connector list[1], so I'm think that it is not supported by > flink community > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ > > > > Best, > > Shammon FY > > > > On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal > wrote: > > Hello Shammon, > > > > Yes socket text stream I am aware of but was thinking if something like as > ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty > <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ > is also supported by Flink? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 27 July 2023 08:15 AM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > There's socket text stream in `DataStream` and you can refer to [1] for > more details. > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program > > > > Best, > > Shammon FY > > > > On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > Does flink provides netty connector for custom TCP source? > > > > Any documentation details please share? > > > > Rgds, > > Kamal > >
RE: Flink netty connector for TCP source
Hello, I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator? Please correct me if above understanding is not correct. Rgds, Kamal From: Hang Ruan Sent: 01 August 2023 08:55 AM To: Kamal Mittal Cc: liu ron ; user@flink.apache.org Subject: Re: Flink netty connector for TCP source Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron mailto:ron9@gmail.com>> Sent: 31 July 2023 10:06 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink? Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 27 July 2023 08:15 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
Re: Flink netty connector for TCP source
Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user 于2023年8月1日周二 10:43写道： > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job Manager rather than at > Task manager? > > > > *From:* liu ron > *Sent:* 31 July 2023 10:06 PM > *To:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi, Kamal > > Currently, we don't provide the connector like TCP source in Flink main > repo. If you need this connector, you can try to implement it refer to the > FLIP-27 source docs > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ > > > > Best, > > Ron > > > > Shammon FY 于2023年7月27日周四 11:23写道： > > I cannot find any information about netty source in flink website and it > is not in the connector list[1], so I'm think that it is not supported by > flink community > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ > > > > Best, > > Shammon FY > > > > On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal > wrote: > > Hello Shammon, > > > > Yes socket text stream I am aware of but was thinking if something like as > ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty > <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ > is also supported by Flink? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 27 July 2023 08:15 AM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > There's socket text stream in `DataStream` and you can refer to [1] for > more details. > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program > > > > Best, > > Shammon FY > > > > On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > Does flink provides netty connector for custom TCP source? > > > > Any documentation details please share? > > > > Rgds, > > Kamal > >
RE: Flink netty connector for TCP source
Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager? From: liu ron Sent: 31 July 2023 10:06 PM To: user@flink.apache.org Subject: Re: Flink netty connector for TCP source Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY mailto:zjur...@gmail.com>> 于2023年7月27日周四 11:23写道： I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-ffc853ff875d9ed3=1=30c3fa9e-7c08-4486-8017-0d78c3c714ac=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink? Rgds, Kamal From: Shammon FY mailto:zjur...@gmail.com>> Sent: 27 July 2023 08:15 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
Re: Flink netty connector for TCP source
Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY 于2023年7月27日周四 11:23写道： > I cannot find any information about netty source in flink website and it > is not in the connector list[1], so I'm think that it is not supported by > flink community > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ > > Best, > Shammon FY > > On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal > wrote: > >> Hello Shammon, >> >> >> >> Yes socket text stream I am aware of but was thinking if something like >> as ‘ >> https://github.com/apache/bahir-flink/tree/master/flink-connector-netty’ >> is also supported by Flink? >> >> >> >> Rgds, >> >> Kamal >> >> >> >> *From:* Shammon FY >> *Sent:* 27 July 2023 08:15 AM >> *To:* Kamal Mittal >> *Cc:* user@flink.apache.org >> *Subject:* Re: Flink netty connector for TCP source >> >> >> >> Hi Kamal, >> >> >> >> There's socket text stream in `DataStream` and you can refer to [1] for >> more details. >> >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program >> >> >> >> Best, >> >> Shammon FY >> >> >> >> On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user < >> user@flink.apache.org> wrote: >> >> Hello, >> >> >> >> Does flink provides netty connector for custom TCP source? >> >> >> >> Any documentation details please share? >> >> >> >> Rgds, >> >> Kamal >> >>
Re: Flink netty connector for TCP source
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/ Best, Shammon FY On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal wrote: > Hello Shammon, > > > > Yes socket text stream I am aware of but was thinking if something like as > ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty’ > is also supported by Flink? > > > > Rgds, > > Kamal > > > > *From:* Shammon FY > *Sent:* 27 July 2023 08:15 AM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: Flink netty connector for TCP source > > > > Hi Kamal, > > > > There's socket text stream in `DataStream` and you can refer to [1] for > more details. > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program > > > > Best, > > Shammon FY > > > > On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello, > > > > Does flink provides netty connector for custom TCP source? > > > > Any documentation details please share? > > > > Rgds, > > Kamal > >
RE: Flink netty connector for TCP source
Hello Shammon, Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty’ is also supported by Flink? Rgds, Kamal From: Shammon FY Sent: 27 July 2023 08:15 AM To: Kamal Mittal Cc: user@flink.apache.org Subject: Re: Flink netty connector for TCP source Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
Re: Flink netty connector for TCP source
Hi Kamal, There's socket text stream in `DataStream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user wrote: > Hello, > > > > Does flink provides netty connector for custom TCP source? > > > > Any documentation details please share? > > > > Rgds, > > Kamal >
Flink netty connector for TCP source
Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal
