Andrew Feng created FLINK-39724:
-----------------------------------
Summary: CsvFormatReader produces a stream which fails get_type()
and assign_timestamps_and_watermarks()
Key: FLINK-39724
URL: https://issues.apache.org/jira/browse/FLINK-39724
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 2.2.0
Reporter: Andrew Feng
Attachments: CsvFormatReader.tar.gz
See good.py and bad.py attached.
The only difference is good.py puts the stream through the identity map. This
is highly counter-intuitive and seems to be undocumented. In v2.2 documentation:
{code:python}
schema = CsvSchema.builder() \
.add_number_column('id', number_type=DataTypes.BIGINT()) \
.add_array_column('array', separator='#', element_type=DataTypes.INT()) \
.set_column_separator(',') \
.build()
source = FileSource.for_record_stream_format(
CsvReaderFormat.for_schema(schema), CSV_FILE_PATH).build()
# the type of record will be Types.ROW_NAMED(['id', 'array'], [Types.LONG(),
Types.LIST(Types.INT())])
ds = env.from_source(source, WatermarkStrategy.no_watermarks(), 'csv-source')
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
