JingsongLi commented on code in PR #10570: URL: https://github.com/apache/paimon/pull/10570#discussion_r4306336421
########## paimon-python/pypaimon/blob/managed_blob_payload.py: ########## @@ -0,0 +1,129 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Resolve primary-key managed BLOB descriptors to payload bytes.""" + +from pypaimon.blob.managed_blob_reference_collector import _map_items +from pypaimon.schema.data_types import ( + is_array_blob_type, + is_blob_type, + is_map_blob_type, + managed_blob_field_names, +) +from pypaimon.table.row.blob import Blob, BlobDescriptor +from pypaimon.utils.path import resolve_path + + +def resolve_table_blob_uri(table_path, uri): + """Join a table-relative pack URI onto the table directory. + + Absolute paths and URIs that already carry a scheme are left unchanged so + descriptors written with a filesystem path still open. + """ + if uri is None or table_path is None: + return uri + normalized = uri.replace('\\', '/') + if (normalized.startswith('/') + or normalized.startswith('file:') + or '://' in normalized): + return uri + return resolve_path(str(table_path), normalized) Review Comment: [P2] Qualify pack URIs before publishing them from relative warehouses A relative local warehouse is retained by FileSystemCatalog. The externalizer now publishes the pack writer's full path, so with warehouse `./wh` the descriptor already contains `./wh/default.db/t/bucket-0/<pack>.managed.blob`. This branch treats it as table-relative and prepends `table_path` again. I reproduced this through public Python APIs on this head with native write/read/scan disabled: create a PK table `(id INT, payload BLOB)`, write `(1, b'visible')`, commit, then `new_read().to_arrow(...)`. Commit succeeds, but read fails trying to open `wh/default.db/t/wh/default.db/t/bucket-0/<pack>.managed.blob`. Please publish a qualified/absolute pack URI for local warehouses (also matching Java's direct-URI contract), or reliably distinguish these complete relative paths from table-relative descriptors. Add the relative-warehouse write → commit → read case. ########## paimon-python/pypaimon/casting/schema_evolution.py: ########## @@ -187,4 +187,47 @@ def cast_array_for_schema_evolution(array, target_type, source_paimon_type=None, if (pa.types.is_signed_integer(target_type) and (pa.types.is_floating(array.type) or pa.types.is_decimal(array.type))): return _numeric_to_integer(array, target_type) - return array.cast(target_type, safe=False) + try: + return array.cast(target_type, safe=False) + except pa.ArrowNotImplementedError: + # PyArrow 6 cannot cast a map or list as a whole when only the child + # binary type changes. Child arrays themselves still cast. + rebuilt = _cast_nested_container(array, target_type) + if rebuilt is None: + raise + return rebuilt + + +def _cast_child(array, target_type): + if array.type == target_type: + return array + try: + return array.cast(target_type, safe=False) + except pa.ArrowNotImplementedError: + return _cast_nested_container(array, target_type) + + +def _cast_nested_container(array, target_type): + """Rebuild a list or map after casting its children.""" + if isinstance(array, pa.ChunkedArray): + chunks = [_cast_nested_container(chunk, target_type) for chunk in array.chunks] + if any(chunk is None for chunk in chunks): + return None + return pa.chunked_array(chunks, type=target_type) + if pa.types.is_list(array.type) and pa.types.is_list(target_type): + values = _cast_child(array.values, target_type.value_type) + if values is None: + return None + return pa.ListArray.from_arrays(array.offsets, values, mask=array.is_null()) + if pa.types.is_large_list(array.type) and pa.types.is_large_list(target_type): + values = _cast_child(array.values, target_type.value_type) + if values is None: + return None + return pa.LargeListArray.from_arrays(array.offsets, values, mask=array.is_null()) + if pa.types.is_map(array.type) and pa.types.is_map(target_type): + keys = _cast_child(array.keys, target_type.key_type) + items = _cast_child(array.items, target_type.item_type) + if keys is None or items is None: + return None + return pa.MapArray.from_arrays(array.offsets, keys, items, mask=array.is_null()) Review Comment: [P2] Use a MAP reconstruction path supported by PyArrow 6 The fallback is specifically needed on older Arrow, but PyArrow 6.0.1's MapArray has no `offsets` attribute. The current Python 3.6 CI job reproduces this in both newly added mixed tests: `test_py_write_managed_blob` and `test_read_python_managed_blob_after_java_gc`. Reading `MAP<STRING,BLOB>` first hits the unsupported `map<string,binary>` → `map<string,large_binary>` cast, then raises `AttributeError` here, so normal managed MAP BLOB reads fail on this supported runtime. See the failed job: https://github.com/apache/paimon/actions/runs/39092744792/job/117572794832 . Please rebuild the nested array using APIs available in the minimum supported PyArrow version, preserving sliced offsets and null maps, and run this mixed regression on that version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
