Re: SDF/Unbounded/Stream tests examples

[Alexey Romanenko]

Please, see my comments inline.

> On 12 Jan 2022, at 20:15, Marco Robles  wrote:
> 
> I am looking for integration tests for SDF implementation (testing the 
> pipeline) for unbounded-stream that may help me as an example to validate 
> some expected behaviors in my current implementation for the SDF tests for 
> PulsarIO connector.

Are you looking for something specific to test? How should this be different 
from other integration tests?

> I was using KafkaIO as an example for the testing, but it seems the SDF 
> version for the testing is not being used (Pipeline tests 
> ).
>  

Actually, both types of Kafka read, SDF or Source API one, can be used within 
KafkaIO.Read (which is used in 
KafkaIOIT.testKafkaIOReadsAndWritesCorrectlyInStreaming() to read messages in 
testing pipeline). Depending on provided options, KafkaIO.Read will expand 
either into ReadFromKafkaViaUnbounded or ReadFromKafkaViaSDF transforms [1] So, 
you need to make sure that you run a test with proper options and it executes 
the right code branch in the end.

[1] 
https://github.com/apache/beam/blob/a3bf36eae08a7e31ec862674180466166b9cf501/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1294

—
Alexey


> 
> Thanks in advance
> 
> -- 
> Marco Robles | WIZELINE
> Software Engineer
> marco.rob...@wizeline.com
>  
> This email and its contents (including any attachments) are being sent to
> you on the condition of confidentiality and may be protected by legal
> privilege. Access to this email by anyone other than the intended recipient
> is unauthorized. If you are not the intended recipient, please immediately
> notify the sender by replying to this message and delete the material
> immediately from your system. Any further use, dissemination, distribution
> or reproduction of this email is strictly prohibited. Further, no
> representation is made with respect to any content contained in this email.

SDF/Unbounded/Stream tests examples

[Marco Robles]

Hi,

I am looking for *integration tests for SDF implementation* (testing the
pipeline) for unbounded-stream that may help me as an *example* to validate
some expected behaviors in my current implementation for the SDF tests for
PulsarIO connector.

I was using KafkaIO as an example for the testing, but it seems the SDF
version for the testing is not being used (Pipeline tests

).

Thanks in advance

-- 

*Marco Robles* *|* WIZELINE

Software Engineer

marco.rob...@wizeline.com

-- 
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*