HyukjinKwon commented on code in PR #59305: URL: https://github.com/apache/spark/pull/59305#discussion_r4237227263
########## sql/api/src/main/scala/org/apache/spark/sql/types/FileType.scala: ########## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.types + +import org.apache.spark.annotation.Unstable + +/** + * The data type representing a reference to an external file, such as a PDF, image, audio or + * video. A value holds the location of the file and a small amount of metadata describing it + * (size, content type and checksum); the bytes of the file are never stored in the value. + * + * In SQL the type is written `FILE EXTERNAL`, and its JSON representation is `"file"`. + * + * The type is physically represented as a struct but deliberately does not extend [[StructType]]. Review Comment: **Nit (P3):** This paragraph describes the end-state design as if it already holds. In this PR, `PhysicalDataType(FileType)` falls through to `UninitializedPhysicalType`, so there is no struct representation yet. `TypeUtils.checkForMapKeyType` only rejects `MapType`/`VariantType` keys, so `map<FILE EXTERNAL, int>` is accepted. `ExprUtils.checkValidGroupingExprs` only rejects `VariantType`, so a FileType grouping expression passes analysis. There is no clustering-column type check. Only the join/sort restriction (through `OrderUtils.isOrderable`) and the partition-column restriction hold today. The PR description lists physical representation and comparability checks as later parts. Could this paragraph describe the struct layout and these restrictions as planned, or add those sentences in the parts that implement them? Enforcement does not need to move into this PR. **Verification:** - **Inspection:** The FileType class comment states only guarantees this part provides, and marks the physical layout and the usage restrictions as planned or omits them. - **Compile:** The sql/api module still compiles, and API doc generation accepts the edited comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
