[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[ https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442810#comment-17442810 ] ASF GitHub Bot commented on DRILL-8027: --- vvysotskyi merged pull request #2357: URL: https://github.com/apache/drill/pull/2357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Format plugin for Apache Iceberg > > > Key: DRILL-8027 > URL: https://issues.apache.org/jira/browse/DRILL-8027 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Labels: plugin > Fix For: 1.20.0 > > > Implement a format plugin for Apache Iceberg. > Plugin should be able to: > - support reading data from Iceberg tables in Parquet, Avro, and ORC formats > - push down fields used in the project > - push down supported filter expressions > - spit and parallelize reading tasks > - provide a way for specifying Iceberg-specific configurations > - read specific snapshot versions if configured > - read table metadata (entries, files, history, snapshots, manifests, > partitions, etc.) > - support schema provisioning -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442797#comment-17442797
]
ASF GitHub Bot commented on DRILL-8027:
---
vvysotskyi commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r748352763
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/ExprToRex.java
##
@@ -62,31 +61,25 @@ public static RelDataTypeField findField(String fieldName,
RelDataType rowType)
return null;
}
- private RexNode makeItemOperator(String[] paths, int index, RelDataType
rowType) {
-if (index == 0) { //last one, return ITEM([0]-inputRef, [1] Literal)
- final RelDataTypeField field = findField(paths[0], rowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-return builder.makeCall(SqlStdOperatorTable.ITEM,
-makeItemOperator(paths, index - 1, rowType),
-builder.makeLiteral(paths[index]));
- }
-
@Override
public RexNode visitSchemaPath(SchemaPath path, Void value) throws
RuntimeException {
-PathSegment.NameSegment rootSegment = path.getRootSegment();
-if (rootSegment.isLastPath()) {
- final RelDataTypeField field = findField(rootSegment.getPath(),
newRowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-List paths = Lists.newArrayList();
-while (rootSegment != null) {
- paths.add(rootSegment.getPath());
- rootSegment = (PathSegment.NameSegment) rootSegment.getChild();
+PathSegment pathSegment = path.getRootSegment();
+
+RelDataTypeField field = findField(pathSegment.getNameSegment().getPath(),
newRowType);
+RexNode rexNode = field == null ? null :
builder.makeInputRef(field.getType(), field.getIndex());
+while (!pathSegment.isLastPath()) {
+ pathSegment = pathSegment.getChild();
+ RexNode ref;
+ if (pathSegment.isNamed()) {
+ref = builder.makeLiteral(pathSegment.getNameSegment().getPath());
+ } else {
+ref =
builder.makeBigintLiteral(BigDecimal.valueOf(pathSegment.getArraySegment().getIndex()));
Review comment:
Calcite API requires BigDecimal
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java
##
@@ -56,7 +59,23 @@
AbstractWriter getWriter(PhysicalOperator child, String location,
List partitionColumns) throws IOException;
- Set getOptimizerRules();
+ @Deprecated
+ default Set getOptimizerRules() {
+return Collections.emptySet();
+ }
+
+ default Set getOptimizerRules(PlannerPhase phase) {
+switch (phase) {
+ case PHYSICAL:
+return getOptimizerRules();
+ case LOGICAL:
+ case JOIN_PLANNING:case LOGICAL_PRUNE_AND_JOIN:
Review comment:
Thanks, fixed.
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/format/IcebergFormatPlugin.java
##
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.iceberg.format;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.AbstractWriter;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.planner.common.DrillStatsTable;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.PluginRulesProviderImpl;
+import org.apache.drill.exec.store.StoragePluginRulesSupplier;
+import org.apache.drill.exec.store.dfs.FileSelection;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442587#comment-17442587
]
ASF GitHub Bot commented on DRILL-8027:
---
dzamo commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r747998477
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/ExprToRex.java
##
@@ -62,31 +61,25 @@ public static RelDataTypeField findField(String fieldName,
RelDataType rowType)
return null;
}
- private RexNode makeItemOperator(String[] paths, int index, RelDataType
rowType) {
-if (index == 0) { //last one, return ITEM([0]-inputRef, [1] Literal)
- final RelDataTypeField field = findField(paths[0], rowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-return builder.makeCall(SqlStdOperatorTable.ITEM,
-makeItemOperator(paths, index - 1, rowType),
-builder.makeLiteral(paths[index]));
- }
-
@Override
public RexNode visitSchemaPath(SchemaPath path, Void value) throws
RuntimeException {
-PathSegment.NameSegment rootSegment = path.getRootSegment();
-if (rootSegment.isLastPath()) {
- final RelDataTypeField field = findField(rootSegment.getPath(),
newRowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-List paths = Lists.newArrayList();
-while (rootSegment != null) {
- paths.add(rootSegment.getPath());
- rootSegment = (PathSegment.NameSegment) rootSegment.getChild();
+PathSegment pathSegment = path.getRootSegment();
+
+RelDataTypeField field = findField(pathSegment.getNameSegment().getPath(),
newRowType);
+RexNode rexNode = field == null ? null :
builder.makeInputRef(field.getType(), field.getIndex());
+while (!pathSegment.isLastPath()) {
+ pathSegment = pathSegment.getChild();
+ RexNode ref;
+ if (pathSegment.isNamed()) {
+ref = builder.makeLiteral(pathSegment.getNameSegment().getPath());
+ } else {
+ref =
builder.makeBigintLiteral(BigDecimal.valueOf(pathSegment.getArraySegment().getIndex()));
Review comment:
Out of curiosity, what is BigDecimal doing here? Why not just an
integer?
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java
##
@@ -56,7 +59,23 @@
AbstractWriter getWriter(PhysicalOperator child, String location,
List partitionColumns) throws IOException;
- Set getOptimizerRules();
+ @Deprecated
+ default Set getOptimizerRules() {
+return Collections.emptySet();
+ }
+
+ default Set getOptimizerRules(PlannerPhase phase) {
+switch (phase) {
+ case PHYSICAL:
+return getOptimizerRules();
+ case LOGICAL:
+ case JOIN_PLANNING:case LOGICAL_PRUNE_AND_JOIN:
Review comment:
Is this meant to be on one line?
##
File path: contrib/format-iceberg/README.md
##
@@ -0,0 +1,117 @@
+# Apache Iceberg format plugin
+
+This format plugin enabled Drill to query Apache Iceberg tables.
+
+Unlike regular format plugins, the Iceberg table is a folder with data and
metadata files, but Drill checks the presence
+of the `metadata` folder to ensure that the table is Iceberg one.
+
+Drill supports reading all formats of Iceberg tables available at this moment:
Parquet, Avro, and ORC.
+No need to provide actual table format, it will be discovered automatically.
+
+For details related to Apache Iceberg table format, please refer to [official
docs](https://iceberg.apache.org/#).
+
+## Supported optimizations and features
+
+### Project pushdown
+
+This format plugin supports project and filter pushdown optimizations.
+
+For the case of project pushdown, only specified in the query columns will be
read, even if it is a nested column. In
Review comment:
```suggestion
For the case of project pushdown, only columns specified in the query will
be read, even they are nested columns. In
```
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/IcebergWork.java
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442377#comment-17442377
]
ASF GitHub Bot commented on DRILL-8027:
---
vvysotskyi commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r747693599
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/format/IcebergFormatPlugin.java
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.iceberg.format;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.AbstractWriter;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.planner.common.DrillStatsTable;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.PluginRulesProviderImpl;
+import org.apache.drill.exec.store.StoragePluginRulesSupplier;
+import org.apache.drill.exec.store.dfs.FileSelection;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FormatMatcher;
+import org.apache.drill.exec.store.dfs.FormatPlugin;
+import org.apache.drill.exec.store.iceberg.IcebergGroupScan;
+import org.apache.drill.exec.store.iceberg.plan.IcebergPluginImplementor;
+import org.apache.drill.exec.store.plan.rel.PluginRel;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+public class IcebergFormatPlugin implements FormatPlugin {
+
+ private static final String ICEBERG_CONVENTION_PREFIX = "ICEBERG.";
+
+ private final FileSystemConfig storageConfig;
+
+ private final IcebergFormatPluginConfig config;
+
+ private final Configuration fsConf;
+
+ private final DrillbitContext context;
+
+ private final String name;
+
+ private final IcebergFormatMatcher matcher;
+
+ private final StoragePluginRulesSupplier storagePluginRulesSupplier;
+
+ public IcebergFormatPlugin(
+String name,
+DrillbitContext context,
+Configuration fsConf,
+FileSystemConfig storageConfig,
+IcebergFormatPluginConfig config) {
+this.storageConfig = storageConfig;
+this.config = config;
+this.fsConf = fsConf;
+this.context = context;
+this.name = name;
+this.matcher = new IcebergFormatMatcher(this);
+this.storagePluginRulesSupplier = storagePluginRulesSupplier(name);
+ }
+
+ private static StoragePluginRulesSupplier storagePluginRulesSupplier(String
name) {
+Convention convention = new Convention.Impl(ICEBERG_CONVENTION_PREFIX +
name, PluginRel.class);
+return StoragePluginRulesSupplier.builder()
+ .rulesProvider(new PluginRulesProviderImpl(convention,
IcebergPluginImplementor::new))
+ .supportsFilterPushdown(true)
+ .supportsProjectPushdown(true)
+ .convention(convention)
+ .build();
+ }
+
+ @Override
+ public boolean supportsRead() {
+return true;
+ }
+
+ @Override
+ public boolean supportsWrite() {
Review comment:
Yep, in the future, we definitely should add it.
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/IcebergGroupScan.java
##
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[ https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442197#comment-17442197 ] ASF GitHub Bot commented on DRILL-8027: --- Z0ltrix commented on a change in pull request #2357: URL: https://github.com/apache/drill/pull/2357#discussion_r747358905 ## File path: pom.xml ## @@ -137,6 +137,7 @@ 1.18.20 0.1.1 0.20 +0.12.0 Review comment: theres a new iceberg release https://github.com/apache/iceberg/releases/tag/apache-iceberg-0.12.1 Maybe we should use it directly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Format plugin for Apache Iceberg > > > Key: DRILL-8027 > URL: https://issues.apache.org/jira/browse/DRILL-8027 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Labels: plugin > Fix For: 1.20.0 > > > Implement a format plugin for Apache Iceberg. > Plugin should be able to: > - support reading data from Iceberg tables in Parquet, Avro, and ORC formats > - push down fields used in the project > - push down supported filter expressions > - spit and parallelize reading tasks > - provide a way for specifying Iceberg-specific configurations > - read specific snapshot versions if configured > - read table metadata (entries, files, history, snapshots, manifests, > partitions, etc.) > - support schema provisioning -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441463#comment-17441463
]
ASF GitHub Bot commented on DRILL-8027:
---
cgivre commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r744179295
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/format/IcebergFormatPlugin.java
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.iceberg.format;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.AbstractWriter;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.planner.common.DrillStatsTable;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.PluginRulesProviderImpl;
+import org.apache.drill.exec.store.StoragePluginRulesSupplier;
+import org.apache.drill.exec.store.dfs.FileSelection;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FormatMatcher;
+import org.apache.drill.exec.store.dfs.FormatPlugin;
+import org.apache.drill.exec.store.iceberg.IcebergGroupScan;
+import org.apache.drill.exec.store.iceberg.plan.IcebergPluginImplementor;
+import org.apache.drill.exec.store.plan.rel.PluginRel;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+public class IcebergFormatPlugin implements FormatPlugin {
+
+ private static final String ICEBERG_CONVENTION_PREFIX = "ICEBERG.";
+
+ private final FileSystemConfig storageConfig;
+
+ private final IcebergFormatPluginConfig config;
+
+ private final Configuration fsConf;
+
+ private final DrillbitContext context;
+
+ private final String name;
+
+ private final IcebergFormatMatcher matcher;
+
+ private final StoragePluginRulesSupplier storagePluginRulesSupplier;
+
+ public IcebergFormatPlugin(
+String name,
+DrillbitContext context,
+Configuration fsConf,
+FileSystemConfig storageConfig,
+IcebergFormatPluginConfig config) {
+this.storageConfig = storageConfig;
+this.config = config;
+this.fsConf = fsConf;
+this.context = context;
+this.name = name;
+this.matcher = new IcebergFormatMatcher(this);
+this.storagePluginRulesSupplier = storagePluginRulesSupplier(name);
+ }
+
+ private static StoragePluginRulesSupplier storagePluginRulesSupplier(String
name) {
+Convention convention = new Convention.Impl(ICEBERG_CONVENTION_PREFIX +
name, PluginRel.class);
+return StoragePluginRulesSupplier.builder()
+ .rulesProvider(new PluginRulesProviderImpl(convention,
IcebergPluginImplementor::new))
+ .supportsFilterPushdown(true)
+ .supportsProjectPushdown(true)
+ .convention(convention)
+ .build();
+ }
+
+ @Override
+ public boolean supportsRead() {
+return true;
+ }
+
+ @Override
+ public boolean supportsWrite() {
Review comment:
Perhaps for a separate PR, but maybe we should discuss adding CTAS
support.
##
File path: contrib/format-iceberg/README.md
##
@@ -0,0 +1,99 @@
+# Apache Iceberg format plugin
+
+This format plugin enabled Drill to query Apache Iceberg tables.
+
+Unlike regular format plugins, the Iceberg table is a folder with data and
metadata files, but Drill checks the presence
+of the `metadata` folder to ensure that the table is Iceberg one.
+
+Drill supports reading all formats of Iceberg tables available at this moment:
Parquet, Avro, and ORC.
+No need to provide actual
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[ https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17439912#comment-17439912 ] ASF GitHub Bot commented on DRILL-8027: --- vvysotskyi commented on pull request #2357: URL: https://github.com/apache/drill/pull/2357#issuecomment-962523023 Hi all, work on this PR is completed, so now it is ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Format plugin for Apache Iceberg > > > Key: DRILL-8027 > URL: https://issues.apache.org/jira/browse/DRILL-8027 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Labels: plugin > Fix For: 1.20.0 > > > Implement a format plugin for Apache Iceberg. > Plugin should be able to: > - support reading data from Iceberg tables in Parquet, Avro, and ORC formats > - push down fields used in the project > - push down supported filter expressions > - spit and parallelize reading tasks > - provide a way for specifying Iceberg-specific configurations > - read specific snapshot versions if configured > - read table metadata (entries, files, history, snapshots, manifests, > partitions, etc.) > - support schema provisioning -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[ https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436531#comment-17436531 ] ASF GitHub Bot commented on DRILL-8027: --- vvysotskyi opened a new pull request #2357: URL: https://github.com/apache/drill/pull/2357 # **Under development, please do not review, use or comment yet** # [DRILL-8027](https://issues.apache.org/jira/browse/DRILL-8027): Format plugin for Apache Iceberg ## Description Format plugin for Apache Iceberg. Functionality: - [x] support reading data from Iceberg tables in Parquet, Avro, and ORC formats - [x] push down fields used in the project - [x] push down supported filter expressions - [x] spit and parallelize reading tasks - [x] provide a way for specifying Iceberg-specific configurations - [x] support schema provisioning - [ ] read specific snapshot versions if configured - [ ] read table metadata (entries, files, history, snapshots, manifests, partitions, etc.) - [ ] additional testing ## Documentation TBA ## Testing TBA closes #2269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Format plugin for Apache Iceberg > > > Key: DRILL-8027 > URL: https://issues.apache.org/jira/browse/DRILL-8027 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Labels: plugin > Fix For: 1.20.0 > > > Implement a format plugin for Apache Iceberg. > Plugin should be able to: > - support reading data from Iceberg tables in Parquet, Avro, and ORC formats > - push down fields used in the project > - push down supported filter expressions > - spit and parallelize reading tasks > - provide a way for specifying Iceberg-specific configurations > - read specific snapshot versions if configured > - read table metadata (entries, files, history, snapshots, manifests, > partitions, etc.) > - support schema provisioning -- This message was sent by Atlassian Jira (v8.3.4#803005)
