This is a conclusion I also came across implementing an application receiving
events via TCP/HTTP.
This also brings the benefit, as already said, that you can use the modern
ecosystem for implementing such things and don't need to so it yourself.
I summarized my conclusions developing my application in that paper:
http://dx.doi.org/10.1145/3579142.3594292
Best regards,
Martin
Am 2. Juli 2023 09:13:14 MESZ schrieb "Jan Lukavský" :
>Hi,
>
>in plain TCP socket, there is no 'state' you can return to when restoring from
>checkpoint. All you can do is reopening the socket. You would have to ensure
>fault-tolerance by some (custom) higher-level protocol you implement,
>including persistent storage, replication, etc. You will soon see you are
>actually implementing a commit-log like Apache Kafka, so it might be better to
>use Kafka directly. If your clients need to send data over plain TCP socket,
>you can receive it over TCP (with a custom TCP server application), store to
>Kafka and use Kafka as a data source for your Flink application.
>
>Best,
>
> Jan
>
>On 7/1/23 05:20, Kamal Mittal wrote:
>>
>> Hello,
>>
>> Thanks for your suggestion but please confirm below.
>>
>> Is it the case that TCP socket source job can’t be restored from last
>> checkpoint?
>>
>> Rgds,
>>
>> Kamal
>>
>> *From:*Jan Lukavský
>> *Sent:* 29 June 2023 06:18 PM
>> *To:* user@flink.apache.org
>> *Subject:* Re: Flink TCP custom source - secured server socket
>>
>> > ... a state backward in (processing) time ...
>> (of course not processing, I meant to say event time)
>>
>> On 6/29/23 14:45, Jan Lukavský wrote:
>>
>> Hi Kamal,
>>
>> you probably have several options:
>>
>> a) bundle your private key and certificate into your Flink
>> application's jar (not recommended, your service's private key
>> will have to be not exactly "private")
>> b) create a service which will provide certificate for your
>> service during runtime (e.g. ACME based or similar)
>>
>> I have a different note, though. Flink (or for that matters any
>> streaming engine, I'm more focused on Apache Beam) heavily relies
>> on the ability of sources to restore a state backward in
>> (processing) time. That is definitely not the case of a plain TCP
>> socket. It is likely you will experience data-loss issues with
>> this solution (regardless of security). This might be okay for
>> you, I just felt it would be good to stress this out.
>>
>> Best,
>>
>> Jan
>>
>> On 6/29/23 12:53, Kamal Mittal via user wrote:
>>
>> Hello Community,
>>
>> I have created TCP stream custom source and reading data from
>> TCP stream source.
>>
>> But this TCP connection needs to be secured i.e. SSL based,
>> query is how to configure/provide certificates via Flink for
>> Client-Server secured TCP connection?
>>
>> Rgds,
>>
>> Kamal
>>
