Thank you very much for doing the release!
On Fri, Mar 10, 2023 at 7:30 PM Danny McCormick via dev
wrote:
> The release has been finalized, thanks everyone!
>
> Thanks,
> Danny
>
> On Fri, Mar 10, 2023 at 2:38 PM Danny McCormick
> wrote:
>
>> We determined that the same issue exists in the 2.45
Thank you Robert for the clarification.
Regards
-
Wilson(Xiaoshuang) Wang
Sr. Software Engineer
On Mon, Mar 13, 2023 at 1:34 PM Robert Bradshaw wrote:
> On Mon, Mar 13, 2023 at 12:32 PM wilsonny...@gmail.com
> wrote:
> >
> > Than
On Mon, Mar 13, 2023 at 12:32 PM wilsonny...@gmail.com
wrote:
>
> Thank you Robert.
>
> Yeah, I agree using flink can work in this case. However, in our environment,
> we are using ray beam runner which will take advantage of Ray framework.
>
> Internally, ray beam runner is using FnApiRunner to
Thank you Robert.
Yeah, I agree using flink can work in this case. However, in our
environment, we are using ray beam runner which will take advantage of Ray
framework.
Internally, ray beam runner is using FnApiRunner to run stages and bundles.
After some code walkthrough, I am wondering how FnAp
Hello Everyone,
https://github.com/apache/beam/pull/25686 is approved and merged. I
describe below its two new Beam project assets for those who know
Kubernetes and terraform and those who don't.
*Short Version (For those who know Kubernetes and terraform)*:
This PR provides:
1. An end-to-end
The FnApiRunner is primarily for tiny jobs (development and testing)
and holds all the data in memory. You'll likely have to run with a
"real" runner to operate over datasets of this size. If you want to
run locally, you can pass --runner=FlinkRunner and (assuming you have
Java installed) it will r
Let me provide more details.
We are running TFX and we specified beam FnApiRunner as the underlying
runner type.
Our dataset is a large amount of HDFS files, each around 200MB and the
total are around 200GB.
When running our TFX code, we saw OOM issue. I assume this is due to Beam
FnApiRunner l
Thank you, Ahmed. This is ready now.
On Fri, Mar 3, 2023 at 9:18 AM Ahmed Abualsaud
wrote:
> Thank you Damon, I left a few comments.
>
> On Fri, Mar 3, 2023 at 11:14 AM Damon Douglas via dev
> wrote:
>
>> Hello Everyone,
>>
>> This PR brings BigQuery Schema transforms in line with the others i
Python Beam direct runner.
Regards
-
Wilson(Xiaoshuang) Wang
Sr. Software Engineer
On Mon, Mar 13, 2023 at 11:29 AM Robert Burke wrote:
> Which direct runner? They are language specific.
>
> On Mon, Mar 13, 2023, 11:27 AM wilsonny.
Which direct runner? They are language specific.
On Mon, Mar 13, 2023, 11:27 AM wilsonny...@gmail.com
wrote:
> Hi guys,
>
> We are trying to run our pipeline using direct runner and the input
> dataset is a large amount of HDFS files (few hundred of GB data)
>
> We experienced OOM issue crash. T
Hi guys,
We are trying to run our pipeline using direct runner and the input dataset
is a large amount of HDFS files (few hundred of GB data)
We experienced OOM issue crash. Then inside the direct runner document, I
realized direct runner loads the whole dataset into the memory.
Is there any way
This is your daily summary of Beam's current high priority issues that may need
attention.
See https://beam.apache.org/contribute/issue-priorities for the meaning and
expectations around issue priorities.
Unassigned P1 Issues:
https://github.com/apache/beam/issues/25675 [Bug]: Reenable
Gr
12 matches
Mail list logo