Hi Team, We are designing an ETL pipeline. Pipeline is invoked when there is a new input file(CSV), and Input file schema(columns) can be dynamic.
WE are using Apache Schema to validate the rows in the InputFile. We are creating a Schema(after reading header row from CSV) at runtime in dofn and storing it in pipelineOptions, So we can use it in pardo(using processcontext) to convert and validate input file strings to Rows. To furthur transform Pcollection<Row> we need to use withSchema() on the Pcollection. The question is, how can we pass the runtime schema (which is in pipelineoptions ) to withSchema() ? How can we use RuntimeContext values in transformations? Please also share the patterns to create a runtime Schema and use it for in a pipeline. Thanks & Regards, Sarath G
