Re: Is there a way (seetings) to limit the number of element per worker machine

Hi Roberts, Thank you. I usually work with the custom worker configuration options I will custom it to low number of cores with large memory and see if it solves my problem Thanks so much, — Eila www.orielresearch.com https://www.meetup.com/Deep-Learning-In-Production Sent from my iPhone >

BQ pipeline fires and error

Hi all. I am initiating the following pipeline to read from table and write to new table and receive the following error: Table is ~60,000 rows input_query = “select * from table" p = beam.Pipeline(options=options) # the first list is the root idx_and_sample[0:1] (p | 'Step 5.1.1: read each

Sorting JSON output by key

Hi all, I have a pipeline that outputs json elements Next, I want to read the JSON elements in order by one of the keys. Any idea how to do it within the pipeline? Thanks, Eila — Eila www.orielesearch.com https://www.meetup.com/Deep-Learning-In-Production Sent from my iPhone

Writing pipeline output to google sheet in google drive

Hello, Is it possible to have the pipeline sink to a google sheet within a specific google drive directory. Something like that: p = beam.Pipeline(options=options) (p | 'Step 1: read file ' >> beam.io.ReadFromText(path/to/file) | 'Step 2: process data ' >> beam.ParDo(get_daata(l])) |

Read a file => process => write to multiple files

Hi all, I am looking for a way to read a large file and generate the following 3 files: 1. extract header 2. extract column #1 from all lines 3. extract column # 2 from all files I use DoFn to extract the values. I am looking for a way to redirect the output to three different files? My thought

Re: GoogleCloudOptions.worker_machine_type = 'n1-highcpu-96'

d options. > p = Pipeline(options=options) > > Alternatively you should be able to just specify --worker_machine_type at > the command line if you're parsing the PipelineOptions from sys.argv. Does > that help? > > Brian > > On Tue, May 12, 2020 at 8:30 AM OrielResearch E

GoogleCloudOptions.worker_machine_type = 'n1-highcpu-96'

Hello, I am trying to check if the setting of the resources are actually being implemented. What will be the right way to do it. *the code is:* GoogleCloudOptions.worker_machine_type = 'n1-highcpu-96' and *the dataflow view is *the following (nothing that reflects the highcpu machine. Please

Re: resources management at worker machine or how to debug hanging execution on worker machine

name: "step-3-2-n1-highcpu-96-5912" project_id: "***" region: "us-central1" step_id: "" } type: "dataflow_step" } severity: "ERROR" timestamp: "2020-05-12T05:34:12.500823Z" } On Mon, May 11, 2020 at 12:52 PM OrielResearch Eil

resources management at worker machine or how to debug hanging execution on worker machine

Hi all, I am trying to run the Kallisto package command on the apache beam worker. Below is a table that describes my steps on the apache beam pipeline code and local compute Debian machine (new machine). I used both of them for debug and comparison. On a local machine, the execution completes

Re: compute engine cap - request for configuration advice

Great. thanks On Fri, May 8, 2020 at 2:33 PM Luke Cwik wrote: > You should set max num workers to set an upper bound on how many will be > used in your job. It will not fail the pipeline when that limit is reached. > > On Fri, May 8, 2020 at 10:03 AM OrielResearch Eila Arich-L

compute engine cap - request for configuration advice

Hi all, I am hitting the compute engine limit. I need to escalate that request with Google support to increase this quota. Could you please advise what is the best way to manage this? see attached the exact quota screenshot. Does limiting the num_of_workers will allow the job to proceed with a

Re: adding apt-get to setup.py fails passing apt-get commands

the Java folder to PATH. ['export', 'PATH=$PATH:/opt/userowned/jdk-14.0.1/bin'] error: [Errno 2] No such file or directory: 'export' What is the right way to use export in setup time? Thanks, Eila On Tue, May 5, 2020 at 12:47 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote

Re: adding apt-get to setup.py fails passing apt-get commands

-zero exit status 100 On Mon, May 4, 2020 at 1:43 PM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi Robert, > > Thank you. My mistake. It is actually not working. > testing it with suborcess.check_call now and will let you know > Best, > Eila > >

Re: adding apt-get to setup.py fails passing apt-get commands

:* stdout_data, *stderr_data* = p.communicate() print('Command output: %s' % stdout_data) *print('Command error data : %s' % stderr_data)* So that issue is resolved for me. Thanks, Eila On Sat, May 2, 2020 at 11:27 PM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi all, &

adding apt-get to setup.py fails passing apt-get commands

Hi all, I have experience and very odd behaviour. when executing the setup.py with the following CUSTOM COMMAND CUSTOM_COMMANDS = [['echo', 'Custom command worked!'], ['apt-get', 'update'], ['apt-get', 'install', '-y', 'unzip']] everything works great. when

Error and initiating new worker when executing a subprocess with Popen

Hi all, I would appreciate any help on that issue. I have no idea where to start debugging that issue. On a local machine, it is working fine. The challenge is to have it working on a worker machine. There is an option to run the command below(kallisto) using multiple threads with -t [#

Executing a third party library command on beam workers failure

Hello, I would like to run a third party library. The library can be copied to the machines or installed via conda. I have tried the following two methods, both of them failed *method 1:* - use the setup.py for wget & tar the library to the workers foler: /opt/userowned/ - execute the command

Re: Copying tar.gz libraries to apache-beam workers

gt; > See the Anaconda silent install[1] instructions for more details. > > 1: https://docs.anaconda.com/anaconda/install/silent-mode/#linux-macos > > > On Fri, Apr 17, 2020 at 9:28 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Than

Re: Copying tar.gz libraries to apache-beam workers

12 PM Luke Cwik wrote: > On Dataflow you should be able to use /opt/userowned > > On Fri, Apr 17, 2020 at 9:01 AM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> See inline >> >> >> — >> Eila >> www.orielesearch.c

Re: Copying tar.gz libraries to apache-beam workers

t should help you locate any errors that might have happened when executing > the custom commands. Will try that. Is it correct to target to this folder. Any other folder is ‘dedicated’ for custom downloads? Thanks Eila > >> On Thu, Apr 16, 2020 at 6:30 PM OrielResearch Eila

Copying tar.gz libraries to apache-beam workers

Hi all, This is a question that I have post on user-h...@apache.org. in case, the other one is not a valid address. posting here again. if you have already received it, apologies for the spam. I hope that you are all well. I would like to copy a tools library into the worker machines and uses

Re: transpose CSV transform

modify this to work on a CSV source. >> >> [1] >> https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-bigquery-transpose >> >> >> On Sun, Jan 13, 2019 at 1:58 AM OrielResearch Eila Arich-Landkof < >> e...@orielresea

INFO:oauth2client.client:Attempting refresh to obtain initial access_token

Hello all, My pipeline looks like the following: (p | "read TXT " >> beam.io.ReadFromText('gs://path/to/file.txt',skip_header_lines= False) | "transpose file " >> beam.ParDo(transposeCSVFn(List1,List2))) The following error is being fired for the read: INFO:oauth2client.client:Attempting

transpose CSV transform

Hi all, I am working with many CSV files where the common part is the row names and therefore, my processing should be by columns. My plan is to have the tables transposed and have the combines tables written into BQ. So , the code should perform: 1. transpose the tables (columns -> new_rows,

Re: Recordings and presentations from Beam Summit London 2018

Thank you for sharing. will definitely watch!!! Happy New Year to all of you, Eila On Thu, Dec 20, 2018 at 6:49 PM Manu Zhang wrote: > Thanks for sharing. The YouTube channel link is > https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ > > Thanks, > Manu Zhang > On Nov 2, 2018, 11:06 PM

Re: 2019 Beam Events

agree  On Tue, Dec 4, 2018 at 5:41 AM Chaim Turkel wrote: > Israel would be nice to have one > chaim > On Tue, Dec 4, 2018 at 12:33 AM Griselda Cuevas wrote: > > > > Hi Beam Community, > > > > I started curating industry conferences, meetups and events that are > relevant for Beam, this

using tfma / ModelAnalysis with tensorflow (not estimator) model

Hello all, I would like to use the modelAnalysis API for model debugging. I don't have a background in model serving and generating the model eval graph for it - so there might be basic background that I am missing. as a start, I would like to add the tfma to this colab (open to other

Re: Advice for piping many CSVs with different columns names to one bigQuery table

ache_beam/examples/complete/game/leader_board.py#L326 > > May I know how many distinct column are you expecting across all files? > > > On Wed, Sep 26, 2018 at 8:06 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi Ankur / users, >> >

Re: Advice for piping many CSVs with different columns names to one bigQuery table

> > Thanks, > Ankur > > > On Tue, Sep 25, 2018 at 12:13 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hello, >> I would like to write large number of CSV file to BQ where the headers >> from all of them is aggregated t

Advice for piping many CSVs with different columns names to one bigQuery table

Hello, I would like to write large number of CSV file to BQ where the headers from all of them is aggregated to one common headers. any advice is very appreciated. The details are: 1. 2.5M CSV files 2. Each CSV file: header of 50-60 columns 2. Each CSV file: one data row there are common columns

Creating a pCollection from large numpy matrix with row and column names

Hello all, I would like to process a large numpy matrix with dimensions: (100K+, 30K+) The column names and the row names are meaningful. My plan was to save the numpy matrix values as a txt file and read it to a PColleciton. However, I am not sure how to add the row names to the element for

Re: INFO:root:Executing Error when executing a pipeline on dataflow

The issue was with the pip version. --download was deprecated. I dont know where this need to be mentioned / fixed. running pip install pip==9.0.3 solved the issue. Thanks, eila On Wed, Aug 22, 2018 at 11:20 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: >

Re: INFO:root:Executing Error when executing a pipeline on dataflow

'--no-binary', ':all:', '--no-deps']' returned non-zero exit status 2 On Wed, Aug 22, 2018 at 10:39 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hello all, > > I am running a pipeline that used to be executed on dataflow with no > issues. I am usi

INFO:root:Executing Error when executing a pipeline on dataflow

Hello all, I am running a pipeline that used to be executed on dataflow with no issues. I am using the datalab environment. See below the error. To my understanding happening before the pipeline code is being is being executed. Any idea what went wrong? Thanks, Eila Executing the pipeline:

Re: Generating data to beam.io.Write(beam.io.BigQuerySink(

et me know if I am writing the data at the right format to BQ (I had no issues writing it to other type of outputs) Thanks for any help, Eila On Mon, Aug 13, 2018 at 1:55 PM, OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > update: > > I tried the following options

Re: Generating data to beam.io.Write(beam.io.BigQuerySink(

, OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hello, > > I am generating a data to be written in new BQ table with a specific > schema. The data is generated at DoFn function. > > My question is: what is the recommended format of data that I should > retur

Generating data to beam.io.Write(beam.io.BigQuerySink(

Hello, I am generating a data to be written in new BQ table with a specific schema. The data is generated at DoFn function. My question is: what is the recommended format of data that I should return from DnFn (getValuesStrFn bellow) ? is it dictionary? list? other? I tried list and str and it

Re: writing to BQ - need help with the error (probably syntax issue)

validate=False, coder=None): > > > On Fri, Aug 10, 2018 at 4:48 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi, >> >> This pipeline is sinking the rows to BQ. I use the following syntax: >> >> | 'write to BQ' >&g

writing to BQ - need help with the error (probably syntax issue)

Hi, This pipeline is sinking the rows to BQ. I use the following syntax: | 'write to BQ' >> beam.io.Write(beam.io.BigQuerySink(dataset='dataset_cell_line.cell_lines_1', schema='accession_list:STRING,comment_list:STRING,derived_from:STRING,disease_list:STRING,name_list:STRING,\

Re: google.cloud.bigQuery version on workers - please HELP

Hi Ahmet, thank you for the detailed explanation. Looking forward for the latest BQ - beam version upgrade. Best, Eila On Fri, Jul 13, 2018 at 9:02 PM, Ahmet Altay wrote: > > > On Thu, Jul 12, 2018 at 7:35 PM, OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wr

Re: google.cloud.bigQuery version on workers - please HELP

equirements.txt is one of those options. > > Ahmet > > [1] https://cloud.google.com/dataflow/docs/concepts/sdk- > worker-dependencies#version-250_1 > > On Thu, Jul 12, 2018 at 8:51 AM, OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >>

INFO:oauth2client.client:Refreshing due to a 401 (attempt 1/2)

Hello, I am running a pipeline that extracts columns from a bigquery table and writes the extracted data to a tsv file. The pipeline is stack on this message: oauth2client.client:Refreshing due to a 401 (attempt 1/2) Could you please let me know what might be the reason for that. is it another

Re: Help with adding python package dependencies when executing pyhton pipeline

ks, Eila On Tue, Jul 3, 2018 at 5:25 PM, OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > thank you. where do i add the reference to requirements.txt? can i do it > from the pipline options code? > > On Tue, Jul 3, 2018 at 5:13 PM, Lukasz Cwik wrote: > >

Re: Help with adding python package dependencies when executing pyhton pipeline

8 at 2:09 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hello all, >> >> >> I am using the python code to run my pipeline. similar to the following: >> >> options = PipelineOptions()google_cloud_options = >> options.view_as

Help with adding python package dependencies when executing pyhton pipeline

Hello all, I am using the python code to run my pipeline. similar to the following: options = PipelineOptions()google_cloud_options = options.view_as(GoogleCloudOptions)google_cloud_options.project = 'my-project-id'google_cloud_options.job_name = 'myjob'google_cloud_options.staging_location =

Re: Pipeline is passing on local runner and failing on Dataflow runner - help with error

failed., A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: label-archs4-tsv-06210931-a4r1-harness-rlqz, label-archs4-tsv-06210931-a4r1-harness-rlqz, label-archs4-tsv-06210931-a4r1-harness-rlqz, label-arch

Pipeline is passing on local runner and failing on Dataflow runner - help with error

Hello, I am running the following pipeline on the local runner with no issues. logging.info('Define the pipeline') p = beam.Pipeline(options=options) samplePath = outputPath ExploreData = (p | "Extract the rows from dataframe" >> beam.io.Read(beam.io.BigQuerySource('archs4.Debug_annotation'))

Re: Returning dataframe from parDo and printing its value - advice?

Thanks!!! On Mon, Jun 18, 2018 at 4:41 PM, Chamikara Jayalath wrote: > A ParDo should always return an iterator not a string. So if you want to > output a single string it should either be "return [str]" or "yield str". > > > On Mon, Jun 18, 2018 at 1:39

Re: Returning dataframe from parDo and printing its value - advice?

eColForSampleFn which takes the 1x164 record and concatenates > each value with ',' in between. > > On Mon, Jun 18, 2018 at 9:00 AM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi, >> >> Is anyone listening on the user@ mailing list? or

Re: Returning dataframe from parDo and printing its value - advice?

oogle Sheet. Thanks, Eila On Fri, Jun 15, 2018 at 2:45 PM, OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi all, > > I am running a pipeline, where a table from BQ is being processed line by > line using ParDo function. > CreateColForSampleFn gene

Returning dataframe from parDo and printing its value - advice?

Hi all, I am running a pipeline, where a table from BQ is being processed line by line using ParDo function. CreateColForSampleFn generates a data frame, with headers and values (shape: 1x164 ) that I want to pass to WriteToText. See the followings: ExploreData = (p | "Extract the rows from

Re: Celebrating Pride... in the Apache Beam Logo

 On Fri, Jun 15, 2018 at 1:50 PM, Griselda Cuevas wrote: > Someone in my team edited some Open-Source-Projects' logos to celebrate > pride and Apache Beam was included! > > > I'm attaching what she did... sprinkling some fun in the mailing list, > because it's Friday! > -- Eila

Re: saving image object on GCS from DoFn

 thank you! On Fri, Apr 27, 2018 at 11:03 AM, Eugene Kirpichov <kirpic...@google.com> wrote: > You need to close the file object, I think that should fix the issue. > > > On Fri, Apr 27, 2018, 7:47 AM OrielResearch Eila Arich-Landkof < > e...@orielresearch.

saving image object on GCS from DoFn

Hello all, I am running the following simplified code from DoFn (ParDo) from PIL import Image img = Image.fromarray(array)img.save('testrgb.png') img.save() with gs:// drive does not work. What would be the recommended way to save the img object on google drive as .png file any advice is

Re: Passing parameter to DoFn in Python

; > def process(self, element): > # use self.samplePath here, will get to remote workers via pickling > > On Tue, Apr 10, 2018 at 4:27 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi all, >> >> Is it possible to pass a s

Passing parameter to DoFn in Python

Hi all, Is it possible to pass a string parameter with DoFn function and what would be the syntax? The call should look something like that: beam.ParDo(SampleFn(samplePath)) how would the class definition be updated? class SampleFn(beam.DoFn): def process(self,element): Thanks, -- Eila

Re: Beam modules are not recognized after dataflow SDK upgrade

nstalling > google-cloud-dataflow on a fresh virtual environment ? > > Thanks, > Cham > > > On Thu, Apr 5, 2018 at 8:51 AM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hello, >> >> It is probably a FAQ question: &

Beam modules are not recognized after dataflow SDK upgrade

Hello, It is probably a FAQ question: After executing: *pip install --upgrade google-cloud-dataflow * The following command fires unrecognized module error: from apache_beam.options.pipeline_options import GoogleCloudOptions Could you please advice what is the way to get rid of that message. I

Re: H5 potential intermediate solution

>> any contributions in this space are welcome. >> >> >>> >>> Best, >>> Eila >>> >>> Sent from my iPhone >>> >>> On Apr 2, 2018, at 7:33 PM, Chamikara Jayalath <chamik...@google.com> >>> wrote: >&g

Re: H5 potential intermediate solution

.WriteToText('gs://archs4/output/', file_name_suffix='.txt')) *Is there a walk around for providing schema for beam.io <http://beam.io>.BigQuerySink?* Many thanks, Eila On Mon, Apr 2, 2018 at 11:33 AM, OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hello all, >

H5 potential intermediate solution

Hello all, I would like to try a different way to leverage Apache beam for H5 => BQ (file to table transfer). For my use case, I would like to read every 10K rows of H5 data (numpy array format), transpose them and write them to BQ 10K columns. 10K is BQ columns limit. My code is below and

Executing a pipeline from datalab - run.wait_until_finished() error

Hello all, When I run the pipeline with 4 samples (very small dataset), I don't get any error on DirectRunner or DataflowRunner When I run it with 50 samples dataset, I get the following error for the run.wait_until_finished() What does this error mean? Thanks, Eila KeyErrorTraceback (most

Dynamic output - Python

Hello all, I would like to print each PCollection element to a different folder on gs://.../ the folder name is not available in advance, but it is available on the pCollection data (or can be calculated prior to printing). My question are: - Is it possible to use the folder path as a parameter

Re: Regarding Beam SlackChannel

thanks!! On Fri, Mar 9, 2018 at 1:58 PM, Lukasz Cwik <lc...@google.com> wrote: > Invite sent, welcome. > > On Thu, Mar 8, 2018 at 7:08 PM, OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Hi Lukasz, >> >> Could you please a

Re: Regarding Beam SlackChannel

Hi Lukasz, Could you please add me as well Thanks, Eila On Thu, Mar 8, 2018 at 2:56 PM, Lukasz Cwik wrote: > Invite sent, welcome. > > On Thu, Mar 8, 2018 at 11:50 AM, Chang Liu wrote: > >> Hello >> >> Can someone please add me to the Beam slackchannel?

Fwd: dataflow HDF5 loading pipeline errors

Hello, Any help will be greatly appreciated!!! I an using dataflow to process H5 (HDF5 format ) file. The H5 file was uploaded to google storage from: https://amp.pharm.mssm. edu/archs4/download.html H5 / HDF5 is an hierarchical data structure to present