rdblue commented on a change in pull request #589: [WIP] Extend metadata with SortOrder URL: https://github.com/apache/incubator-iceberg/pull/589#discussion_r351391663
########## File path: api/src/main/java/org/apache/iceberg/SortField.java ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iceberg; + +import java.io.Serializable; +import java.util.Arrays; +import java.util.Objects; +import org.apache.iceberg.SortTransforms.SortTransform; + +public class SortField implements Serializable { + public enum Direction { + ASC, DESC + } + + public enum NullOrder { + NULLS_FIRST, NULLS_LAST + } + + private final String name; + private final Direction direction; + private final NullOrder nullOrder; + private final SortTransform transform; Review comment: Iceberg should support configuring an order that gets applied to all jobs writing to a table. We want to enable building services that can make changes to a table, like the file format, compression, or sort order used. Embedding the sort order in a job requires having a human change the job and not just table configuration because it requires editing SQL or rewriting a DataFrame operation. Sort order should be declarative, not imperative. You're right that there are times want to have control over a write in a job. For example, you may want to write unsorted data to meet an SLA, but then have a service go rewrite that data with a more optimal sort order. I think this need is less common and the default should be to use the sort order from the table's configuration. We just need a way to override the table configuration. Also, table configuration should support both local and global ordering. Our best practice is to use a global ordering to handle skew, prepare data for filters, and make the data encode or compress better. But, maybe this should be configured separately, like Hive's `DISTRIBUTE BY` and `SORT BY`. What do you think? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
