Re: Problem in jdbc connector with autoincrement value

[Juan Romero]

Thanks!! It was useful!!!

On Sat, 24 Feb 2024 at 6:13 PM XQ Hu via user  wrote:

> Here is what I did:
>
> CREATE TABLE IF NOT EXISTS test2 (id bigint DEFAULT
> nextval('sap_tm_customer_id_seq'::regclass) NOT NULL, name VARCHAR(10),
> load_date_time TIMESTAMP)
>
> make sure id cannot be NULL (you might not need this).
>
> I tried this for my data without using the id field:
>
> class ExampleRow(NamedTuple):
> name: str
> load_date_time: str
>
> For the statement, I used this:
>
> "INSERT INTO test2 VALUES(DEFAULT, ?,?::timestamp)"
>
> DEFAULT fills in the id using sap_tm_customer_id_seq.
>
> I hope this is what you are looking for.
>
>
> On Mon, Feb 19, 2024 at 5:57 PM Juan Romero  wrote:
>
>> Hi guys. I have a table in apache beam that has an auto increment id with
>> a sequence.
>>
>>
>>
>>
>>
>>
>> *CREATE SEQUENCE sap_tm_customer_id_seq;CREATE TABLE IF NOT EXISTS test
>> (   id bigint DEFAULT nextval('sap_tm_customer_id_seq'::regclass),   name
>> VARCHAR(10),   load_date_time TIMESTAMP);*
>>
>> And i have the following pipeline to make the insert into the table:
>> with beam.Pipeline() as p:
>>   _ = (
>>   p
>>   | beam.Create(
>> [
>>
>> ExampleRow(id=1,name='adsf', load_date_time='2023-04-05
>> 12:34:55'),
>> ExampleRow(id=3,name='adsf', load_date_time='2023-04-05
>> 12:34:56')
>> ]).with_output_types(ExampleRow)
>>   | 'Write to jdbc' >> WriteToJdbc(
>>   driver_class_name='org.postgresql.Driver',
>>   jdbc_url='jdbc:postgresql://localhost:5432/postgres',
>>   username='postgres',
>>   password='postgres',
>>   table_name= 'test',
>>   connection_properties="stringtype=unspecified",
>>   statement='INSERT INTO test \
>> VALUES(?,?,?) '
>>   ))
>>
>> The problem is that I haven't been able to insert the register
>> omitting the id field into the pcollection in a way the database
>> automatically assigns the auto increment id.
>>
>> Can you help me? I have spent a lot of time but i have not realize the
>> solution.
>>
>> Thanks!!
>>
>>
>>
>>

Re: Cross Language Runtime error python-Java

[XQ Hu via user]

Great, you figured it out. Thanks for posting this back to the list.

On Sat, Feb 24, 2024 at 5:23 PM George Dekermenjian 
wrote:

> Adding the following to both Dockerfile.launcher and Dockerfile.worker did
> the trick for me.
>
>
>
> COPY --from=apache/beam_java11_sdk:latest /opt/apache/beam/jars
> /opt/apache/beam/jars COPY --from=apache/beam_java11_sdk:latest
> /opt/java/openjdk /opt/java/openjdk ENV JAVA_HOME=/opt/java/openjdk ENV
> PATH="${JAVA_HOME}/bin:${PATH}"
>
>
>
> On Sat, Feb 24, 2024 at 21:55 XQ Hu via user  wrote:
>
>> Does your code work without the launcher? Better check this step by step
>> to figure out which part causes this error.
>>
>> On Sat, Feb 24, 2024 at 3:25 AM George Dekermenjian 
>> wrote:
>>
>>> I have a python pipeline that uses the bigquery storage write method
>>> (cross language with Java). I’m building launcher and worker docker images
>>> and then launching the flex template. The launcher fails due to the
>>> following runtime error in dataflow.
>>>
>>> I’m using runner v2 and it is a streaming pipeline using the streaming
>>> engine.
>>>
>>> Any ideas of what is causing this?
>>>
>>> RuntimeError:
>>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.ClassCastException: class
>>> org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class
>>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger
>>> (org.apache.beam.sdk.transforms.windowing.Repeatedly and
>>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed
>>> module of loader 'app')"
>>> line: "exec.go:66"
>>>
>>>

Re: Problem in jdbc connector with autoincrement value

[XQ Hu via user]

Here is what I did:

CREATE TABLE IF NOT EXISTS test2 (id bigint DEFAULT
nextval('sap_tm_customer_id_seq'::regclass) NOT NULL, name VARCHAR(10),
load_date_time TIMESTAMP)

make sure id cannot be NULL (you might not need this).

I tried this for my data without using the id field:

class ExampleRow(NamedTuple):
name: str
load_date_time: str

For the statement, I used this:

"INSERT INTO test2 VALUES(DEFAULT, ?,?::timestamp)"

DEFAULT fills in the id using sap_tm_customer_id_seq.

I hope this is what you are looking for.


On Mon, Feb 19, 2024 at 5:57 PM Juan Romero  wrote:

> Hi guys. I have a table in apache beam that has an auto increment id with
> a sequence.
>
>
>
>
>
>
> *CREATE SEQUENCE sap_tm_customer_id_seq;CREATE TABLE IF NOT EXISTS test (
>  id bigint DEFAULT nextval('sap_tm_customer_id_seq'::regclass),   name
> VARCHAR(10),   load_date_time TIMESTAMP);*
>
> And i have the following pipeline to make the insert into the table:
> with beam.Pipeline() as p:
>   _ = (
>   p
>   | beam.Create(
> [
>
> ExampleRow(id=1,name='adsf', load_date_time='2023-04-05
> 12:34:55'),
> ExampleRow(id=3,name='adsf', load_date_time='2023-04-05
> 12:34:56')
> ]).with_output_types(ExampleRow)
>   | 'Write to jdbc' >> WriteToJdbc(
>   driver_class_name='org.postgresql.Driver',
>   jdbc_url='jdbc:postgresql://localhost:5432/postgres',
>   username='postgres',
>   password='postgres',
>   table_name= 'test',
>   connection_properties="stringtype=unspecified",
>   statement='INSERT INTO test \
> VALUES(?,?,?) '
>   ))
>
> The problem is that I haven't been able to insert the register
> omitting the id field into the pcollection in a way the database
> automatically assigns the auto increment id.
>
> Can you help me? I have spent a lot of time but i have not realize the
> solution.
>
> Thanks!!
>
>
>
>

Re: Cross Language Runtime error python-Java

[George Dekermenjian]

Adding the following to both Dockerfile.launcher and Dockerfile.worker did
the trick for me.



COPY --from=apache/beam_java11_sdk:latest /opt/apache/beam/jars
/opt/apache/beam/jars COPY --from=apache/beam_java11_sdk:latest
/opt/java/openjdk /opt/java/openjdk ENV JAVA_HOME=/opt/java/openjdk ENV
PATH="${JAVA_HOME}/bin:${PATH}"



On Sat, Feb 24, 2024 at 21:55 XQ Hu via user  wrote:

> Does your code work without the launcher? Better check this step by step
> to figure out which part causes this error.
>
> On Sat, Feb 24, 2024 at 3:25 AM George Dekermenjian 
> wrote:
>
>> I have a python pipeline that uses the bigquery storage write method
>> (cross language with Java). I’m building launcher and worker docker images
>> and then launching the flex template. The launcher fails due to the
>> following runtime error in dataflow.
>>
>> I’m using runner v2 and it is a streaming pipeline using the streaming
>> engine.
>>
>> Any ideas of what is causing this?
>>
>> RuntimeError:
>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>> java.lang.ClassCastException: class
>> org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class
>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger
>> (org.apache.beam.sdk.transforms.windowing.Repeatedly and
>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed
>> module of loader 'app')"
>> line: "exec.go:66"
>>
>>

Re: Problem with pikcler

[Juan Romero]

Thanks!!’

On Sat, 24 Feb 2024 at 12:23 PM Valentyn Tymofieiev via user <
user@beam.apache.org> wrote:

> You might need to pass the --save_main_session pipeline option. See:
> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
>
> Alternatively,  structure your pipeline as a package:
> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython
>
>
> On Fri, Feb 23, 2024 at 6:14 PM Juan Romero  wrote:
>
>> Hi guys.
>>
>> Currently i have the class SaptmCustomerUpsertPipeline that inherited
>> from the GenericPipeline class. Locally the pipeline works fine, but when i
>> try to run it in dataflow i got this error:
>>
>> severity: "ERROR"
>> textPayload: "Error message from worker: generic::unknown: Traceback
>> (most recent call last): File
>> "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
>> line 285, in loads return dill.loads(s) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
>> return load(file, ignore, **kwds) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
>> return Unpickler(file, ignore=ignore, **kwds).load() File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
>> obj = StockUnpickler.load(self) File
>> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in
>> find_class return StockUnpickler.find_class(self, module, name)
>> AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on
>> > '/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
>>
>> Seems that there is an error in the serialization of the child. class.
>> Someone has faced with this problem? . The pipeline works fine with the
>> direct runner.
>>
>>

Re: Cross Language Runtime error python-Java

[XQ Hu via user]

Does your code work without the launcher? Better check this step by step to
figure out which part causes this error.

On Sat, Feb 24, 2024 at 3:25 AM George Dekermenjian 
wrote:

> I have a python pipeline that uses the bigquery storage write method
> (cross language with Java). I’m building launcher and worker docker images
> and then launching the flex template. The launcher fails due to the
> following runtime error in dataflow.
>
> I’m using runner v2 and it is a streaming pipeline using the streaming
> engine.
>
> Any ideas of what is causing this?
>
> RuntimeError:
> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.ClassCastException: class
> org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class
> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger
> (org.apache.beam.sdk.transforms.windowing.Repeatedly and
> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed
> module of loader 'app')"
> line: "exec.go:66"
>
>

Re: Query about `JdbcIO`

[XQ Hu via user]

I did not find BEAM-13846 but this suggests String is never supported:

https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java#L59

However, you could use the code from the test to create yours.

On Thu, Feb 22, 2024 at 11:20 AM Vardhan Thigle via user <
user@beam.apache.org> wrote:

> Hi,
> I had a small query about `JdbcIO`.
> As per the documentation
> 
>  `readWithPartitions` is supported for  Long, DateTime
> 
> , String types for the partition column but on top of the tree code, 
> `PRESET_HELPERS`
> (ref
> )
> support only Long and DateTime.
>
> Was the support for `String` rolled back? If yes could you please help me
> with the exact problem that caused the rollback (or any pointers to a
> previous Issue)?
>
> Regards and Thanks,
> Vardhan Thigle,
> +919535346204 <+91%2095353%2046204>
>
> Regards and Thanks,
> Vardhan Thigle,
> +919535346204 <+91%2095353%2046204>
>

Re: Problem with pikcler

[Valentyn Tymofieiev via user]

You might need to pass the --save_main_session pipeline option. See:
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session

Alternatively,  structure your pipeline as a package:
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython


On Fri, Feb 23, 2024 at 6:14 PM Juan Romero  wrote:

> Hi guys.
>
> Currently i have the class SaptmCustomerUpsertPipeline that inherited from
> the GenericPipeline class. Locally the pipeline works fine, but when i try
> to run it in dataflow i got this error:
>
> severity: "ERROR"
> textPayload: "Error message from worker: generic::unknown: Traceback
> (most recent call last): File
> "/usr/local/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py",
> line 285, in loads return dill.loads(s) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 275, in loads
> return load(file, ignore, **kwds) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 270, in load
> return Unpickler(file, ignore=ignore, **kwds).load() File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 472, in load
> obj = StockUnpickler.load(self) File
> "/usr/local/lib/python3.9/site-packages/dill/_dill.py", line 462, in
> find_class return StockUnpickler.find_class(self, module, name)
> AttributeError: Can't get attribute 'SaptmCustomerUpsertPipeline' on
>  '/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py'>
>
> Seems that there is an error in the serialization of the child. class.
> Someone has faced with this problem? . The pipeline works fine with the
> direct runner.
>
>

Cross Language Runtime error python-Java

[George Dekermenjian]

I have a python pipeline that uses the bigquery storage write method (cross
language with Java). I’m building launcher and worker docker images and
then launching the flex template. The launcher fails due to the following
runtime error in dataflow.

I’m using runner v2 and it is a streaming pipeline using the streaming
engine.

Any ideas of what is causing this?

RuntimeError:
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.ClassCastException: class
org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class
org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger
(org.apache.beam.sdk.transforms.windowing.Repeatedly and
org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed
module of loader 'app')"
line: "exec.go:66"