Re: [EXTERNAL] [EXTERNAL]

[Raphael Sanamyan]

Hello!

I've made a PR[1] and created a task[2] in Jira. Could someone please review 
the PR?

Thanks,
Raphael

[1] 
https://github.com/akvelon/beam/pull/17
[2] 
https://issues.apache.org/jira/browse/BEAM-12511

Re: [EXTERNAL] Re: [EXTERNAL]

[Raphael Sanamyan]

Hello,

Is it somehow related to this work [1]?

No, this work adds the ability to return values from a sql insert query. There 
are no improvements to work with row and schema in it.

Not sure that I got it. Could you elaborate a bit on this?

When we using "Write" with table and without statement, "Write.expand" is 
called, which automatically generates statement and provides input to 
"WriteVoid.expand", but when we using "Write.withResults", only 
"WriteVoid.expand" is called, which can't automatically generate statement. If 
we add conditions there similar to those in "Write.Expand" and move the 
statement generation in "WriteVoid.expand", we'll fix this case

I analyzed the Write class again and it seems to be the only case where there 
is no full support for automatic work with "row". I think it makes sense to 
delete 
todo
 and close the the task, to 
not confuse people. And create a task, to solve this case. What do you think 
about that?


Thanks,
Raphael.

Re: [EXTERNAL] Re:

[Raphael Sanamyan]

Hello,

Here is a case where you need to have a statement and a preparedStatementSetter.

  PCollection dataCollection = pipeline.apply(Create.of(data));
  PCollection rowsWritten =
  dataCollection.apply(
  JdbcIO.write()
  .withDataSourceConfiguration(DATA_SOURCE_CONFIGURATION)
  .withBatchSize(10L)
  .withTable(firstTableName)
  .withResults());
  .dataCollection .
  .apply(Wait.on(rowsWritten))
  .apply(
  JdbcIO.write()
  .withDataSourceConfiguration(DATA_SOURCE_CONFIGURATION)
  .withBatchSize(10L)
  .withTable(secondTableName));

  .run();

In this case, we write data to one table and then to the other, but only after 
the window of data has been fully written to the first table. It is not 
possible to do this with the existing JdbcIO.Write functionality.

Another option for this specific case could be extending the existing class 
instead of adding a schemaApi-specific class. We can add additional conditions 
and move some functionality from Write to WriteVoid to infer beamScheama. What 
do you think about these options?

Schema Providers is not very well documented in Beam, and a bit confusing us. 
We using Beam row as a common abstraction in Beam pipelines, which really meets 
our requirements. Looking to Beam docs/code we saw SchemaProviders for some 
IOs. Those providers seem like wrappers around IOs that help work with schemas 
and conversion data to Beam Rows. Сould you please clarify this a little? If we 
want to improve Beam Schema API what is the architecture-right way to do that?


Thank you,
Raphael.

От: Brian Hulette 
Отправлено: 9 июня 2021 г. 19:12:41
Кому: dev
Копия: Reuven Lax; pabl...@google.com; Ilya Kozyrev
Тема: [EXTERNAL] Re:

> And also the ticket and "// TODO: BEAM-10396 use writeRows() when it's 
> available" appeared later than this functionality was added to "JdbcIO.Write".

Note that this TODO has been moved around through a few refactors. It was 
initially added last summer [1].
You're right that JdbcIO.Write's statement generation functionality was added 
about a year before that [2]. It's possible that the author of [1] didn't 
realize [2] was done. Or maybe there's some reason why it doesn't work there?

+1 for Alexey's requests:
- Identify cases where statement generation in JdbcIO.Write is insufficient, if 
they exist (e.g. can we just use it where that TODO is [3]? If not what goes 
wrong?).
- Update documentation to avoid this confusion in the future.

Brian

[1] https://github.com/apache/beam/pull/12145
[2] https://github.com/apache/beam/pull/8962
[3] https://github.com/apache/beam/pull/14954#discussion_r648456230

On Wed, Jun 9, 2021 at 7:49 AM Alexey Romanenko 
mailto:aromanenko@gmail.com>> wrote:
Hello Raphael,

On 9 Jun 2021, at 09:31, Raphael Sanamyan 
mailto:raphael.sanam...@akvelon.com>> wrote:

The "JdbcIO.Write" allows you to write rows without a statement or statement 
preparer, but not all functionality works without them.

Could you show a use case when the current functionality is not enough?


The method "WithResults" requires a statement and statement preparer. And also 
the ticket<https://issues.apache.org/jira/browse/BEAM-10396> and "// TODO: 
BEAM-10396 use writeRows() when it's 
available"<https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java#L142>
 appeared later than this functionality was added to "JdbcIO.Write". And 
without reading the code, just the documentation, it's not clear that the 
schema is enough.

Agree but the documentation can be updated. On the oath hand, it would be great 
to have some examples that show the needs of WriteRows.

Thanks,
Alexey

Thank you,
Raphael.





От: Pablo Estrada mailto:pabl...@google.com>>
Отправлено: 7 июня 2021 г. 22:43:24
Кому: dev; Reuven Lax
Копия: Ilya Kozyrev
Тема: Re:

*** This Message Is From an External Sender ***
+Reuven Lax<mailto:re...@google.com> do you know if this is already supported 
or not?
I have been able to use `JdbcIO.write()` without specifying a statement nor a 
statement preparer. Is that not what's necessary? I've done this with a named 
class with schemas (i.e. not Row) - is this perhaps the difference?
Best
-P.

On Fri, Jun 4, 2021 at 3:44 PM Robert Bradshaw 
mailto:rober...@google.com>> wrote:
That would be great! I don't know much about this particular issue,
but tips for getting started in general can be found at
https://beam.apache.org/contribute/

On Thu, Jun 3, 2021 at 10:55 AM Raphael Sanamyan
mailto:raphael.sanam...@akvelon.com>> wrote:
>
> Hi, community,
>
> I would like to start work on this task  beam-10396, I

Re: Re:

[Raphael Sanamyan]

Hello Pablo!
The "JdbcIO.Write" allows you to write rows without a statement or statement 
preparer, but not all functionality works without them. The method 
"WithResults" requires a statement and statement preparer. And also the 
ticket<https://issues.apache.org/jira/browse/BEAM-10396> and "// TODO: 
BEAM-10396 use writeRows() when it's 
available"<https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java#L142>
 appeared later than this functionality was added to "JdbcIO.Write". And 
without reading the code, just the documentation, it's not clear that the 
schema is enough.Thank you,
Raphael.





От: Pablo Estrada 
Отправлено: 7 июня 2021 г. 22:43:24
Кому: dev; Reuven Lax
Копия: Ilya Kozyrev
Тема: Re:


*** This Message Is From an External Sender ***

+Reuven Lax<mailto:re...@google.com> do you know if this is already supported 
or not?
I have been able to use `JdbcIO.write()` without specifying a statement nor a 
statement preparer. Is that not what's necessary? I've done this with a named 
class with schemas (i.e. not Row) - is this perhaps the difference?
Best
-P.

On Fri, Jun 4, 2021 at 3:44 PM Robert Bradshaw 
mailto:rober...@google.com>> wrote:
That would be great! I don't know much about this particular issue,
but tips for getting started in general can be found at
https://beam.apache.org/contribute/

On Thu, Jun 3, 2021 at 10:55 AM Raphael Sanamyan
mailto:raphael.sanam...@akvelon.com>> wrote:
>
> Hi, community,
>
> I would like to start work on this task  beam-10396, I hope nobody minds?
> Also, if anyone has any details or developments on this task, I would be glad 
> if you could share them.
>
> Thank you,
> Raphael.
>
>

[no subject]

[Raphael Sanamyan]

Hi, community,

I would like to start work on this task  
beam-10396,
 I hope nobody minds?
Also, if anyone has any details or developments on this task, I would be glad 
if you could share them.

Thank you,
Raphael.

Re: Migrate S3FileSystem

[Raphael Sanamyan]

Hello Ismaël,


I have finished the task "Migrate S3FileSystem to AWS SDK for Java 2" and made 
the PR https://github.com/apache/beam/pull/13914. Could you please review this 
PR or suggest somebody who could do it?


Thank you,

Raphael.


____
От: Raphael Sanamyan
Отправлено: 29 января 2021 г. 1:44:17
Кому: dev@beam.apache.org
Копия: Ilya Kozyrev
Тема: Re: Migrate S3FileSystem


Hello Ismaël,


Thank you for such a quick response. If the main task is to adapt the beam 
classes to the new AWS API, then I have no questions and I will start the task 
and send out a PR for the review soon.

Thank you,
Raphael.



От: Ismaël Mejía 
Отправлено: 28 января 2021 г. 15:37:04
Кому: dev
Тема: Re: Migrate S3FileSystem

Hello Raphael,

You don't need to change the version of the SDK because at the moment
we do support AWS SDK for Java 2, you just have to put the classes in
the correct module.
https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2
[https://avatars.githubusercontent.com/u/47359?s=400=4]<https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2>

beam/sdks/java/io/amazon-web-services2 at master · apache/beam · 
GitHub<https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2>
github.com
Apache Beam is a unified programming model for Batch and Streaming - apache/beam



The expected outcome is just to reproduce what S3FileSystem.java does
for the amazon-web-services module, Main task is to adapt the Beam
classes to the new AWS API

If more doubts don't hesitate to ask.

Best,
Ismaël

On Thu, Jan 28, 2021 at 11:38 AM Raphael Sanamyan
 wrote:
>
> Hi, community,
>
>
> I'm going to implement a task "Migrate S3FileSystem to AWS SDK for Java 2". 
> I'm planning to change the version of SDK to the new one and to fix troubles 
> in case they appear. If anyone has any details of this task, it would be nice 
> if you share them, since there are no definite requirements and comments at 
> the task's description.
>
>
> Thank you,
>
> Raphael.

Re: Migrate S3FileSystem

[Raphael Sanamyan]

Hello Ismaël,


Thank you for such a quick response. If the main task is to adapt the beam 
classes to the new AWS API, then I have no questions and I will start the task 
and send out a PR for the review soon.

Thank you,
Raphael.



От: Ismaël Mejía 
Отправлено: 28 января 2021 г. 15:37:04
Кому: dev
Тема: Re: Migrate S3FileSystem

Hello Raphael,

You don't need to change the version of the SDK because at the moment
we do support AWS SDK for Java 2, you just have to put the classes in
the correct module.
https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2
[https://avatars.githubusercontent.com/u/47359?s=400=4]<https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2>

beam/sdks/java/io/amazon-web-services2 at master · apache/beam · 
GitHub<https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2>
github.com
Apache Beam is a unified programming model for Batch and Streaming - apache/beam



The expected outcome is just to reproduce what S3FileSystem.java does
for the amazon-web-services module, Main task is to adapt the Beam
classes to the new AWS API

If more doubts don't hesitate to ask.

Best,
Ismaël

On Thu, Jan 28, 2021 at 11:38 AM Raphael Sanamyan
 wrote:
>
> Hi, community,
>
>
> I'm going to implement a task "Migrate S3FileSystem to AWS SDK for Java 2". 
> I'm planning to change the version of SDK to the new one and to fix troubles 
> in case they appear. If anyone has any details of this task, it would be nice 
> if you share them, since there are no definite requirements and comments at 
> the task's description.
>
>
> Thank you,
>
> Raphael.

Migrate S3FileSystem

[Raphael Sanamyan]

Hi, community,

I'm going to implement a task "Migrate S3FileSystem to AWS SDK for Java 
2". I'm planning to change the 
version of SDK to the new one and to fix troubles in case they appear. If 
anyone has any details of this task, it would be nice if you share them, since 
there are no definite requirements and comments at the task's description.


Thank you,

Raphael.

Bug with PubsubIO and FixedWindows

[Raphael Sanamyan]

Hi community,

I would like to start work on this bug 
beam-7853,
 I hope nobody minds?
Also, if anyone has any details or developments on this bug, I would be glad if 
you could share them.

Thanks you,
Raphael.

Contributor permission for Beam Jira

[Raphael Sanamyan]

Hi, I am Raphael from Akvelon.

I would like to be a beam contributor.
Could you please add me as a contributor to Beam's Jira issue tracker? I would 
like to create/assign tickets?

My Jira username: raphael.sanamyan

Thanks,
Raphael