Oh that's very interesting! I have a few comments, but we could end up with a
new feature for the Go SDK.
As you've noted, you shouldn't really be manually spinning up step 4. It's up
to the runner to do that, but it does look like for your usage, some assistance
is needed.
The Python Boot
unfortunately didn't work.
I added the @DefaultCoder as you told me and i removed the .withCoder from my
JdbcIO pipe and showed the same error:
java.lang.IllegalStateException: Unable to infer a coder for JdbcIO.readAll()
transform. Provide a coder via withCoder, or ensure that one can be
Well, perhaps it’s a bug (I’ll try to check it on my side later).
Can you try to annotate InboundData class with “@DefaultCoder(AvroCoder.class)”
for example and see if it will work?
> On 23 Mar 2023, at 15:13, Juan Cuzmar wrote:
>
> Alexey yes! here it is:
>
>
>import lombok.*;
>
Thank you, Robert, for your detailed response and the resources you shared.
One thing that I didn't mention is that my goal is to move the setup to EKS
after completing local experimentation. As you pointed out LOOPBACK is
mainly for local setups and testing. I also started with the DOCKER mode,
Alexey yes! here it is:
import lombok.*;
import lombok.extern.jackson.Jacksonized;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
@Jacksonized
@Builder
@Getter
@Value
@EqualsAndHashCode
@ToString
public class
Could you share a class declaration of your InboundData class? Is it just a
POJO?
—
Alexey
> On 23 Mar 2023, at 08:16, Juan Cuzmar wrote:
>
> Hello all,
>
> I hope this message finds you well. I am currently working with Apache Beam's
> JdbcIO and need some guidance regarding setting a
Hi,
I have a pipeline that reads filepaths from pubsub and sends them to avroIO to
parse the actual files and writes them back into parquet.
PubSubIO(filepaths) > AvroIO.parseFilesGenericRecords() > Window >
FileWriter
Each file can contain millions of records which create a
Hello all,
I hope this message finds you well. I am currently working with Apache Beam's
JdbcIO and need some guidance regarding setting a coder for the input data
without resorting to the deprecated withCoder method. I've been trying to
resolve this issue and would appreciate any insights or
