AnandInguva commented on code in PR #30146:
URL: https://github.com/apache/beam/pull/30146#discussion_r1478550067
##########
sdks/python/apache_beam/ml/transforms/handlers.py:
##########
@@ -447,22 +409,22 @@ def expand(
raw_data_metadata = metadata_io.read_metadata(
os.path.join(self.artifact_location, RAW_DATA_METADATA_DIR))
- keyed_raw_data = (raw_data | beam.ParDo(_ComputeAndAttachUniqueID()))
+ keyed_raw_data = (raw_data) # | beam.ParDo(_ComputeAndAttachUniqueID()))
feature_set = [feature.name for feature in
raw_data_metadata.schema.feature]
- keyed_columns_not_in_schema = (
- keyed_raw_data
- | beam.ParDo(_GetMissingColumns(feature_set)))
+ self.data_coder.set_unused_columns(exclude_columns=feature_set)
# To maintain consistency by outputting numpy array all the time,
# whether a scalar value or list or np array is passed as input,
# we will convert scalar values to list values and TFT will ouput
# numpy array all the time.
+ raw_data_list = (
Review Comment:
yes, it is modified. I removed raw from the variable name.
_list: we convert the scalar element to list (len:1) to maintain uniformity.
Users can pass list/np arrays to TFT ops and TFT outputs numpy arrays. Users
when pass scalars, TFT outputs scalars. to maintain consistent output format,
we convert scalar to list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
