[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442587#comment-17442587
]
ASF GitHub Bot commented on DRILL-8027:
---
dzamo commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r747998477
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/ExprToRex.java
##
@@ -62,31 +61,25 @@ public static RelDataTypeField findField(String fieldName,
RelDataType rowType)
return null;
}
- private RexNode makeItemOperator(String[] paths, int index, RelDataType
rowType) {
-if (index == 0) { //last one, return ITEM([0]-inputRef, [1] Literal)
- final RelDataTypeField field = findField(paths[0], rowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-return builder.makeCall(SqlStdOperatorTable.ITEM,
-makeItemOperator(paths, index - 1, rowType),
-builder.makeLiteral(paths[index]));
- }
-
@Override
public RexNode visitSchemaPath(SchemaPath path, Void value) throws
RuntimeException {
-PathSegment.NameSegment rootSegment = path.getRootSegment();
-if (rootSegment.isLastPath()) {
- final RelDataTypeField field = findField(rootSegment.getPath(),
newRowType);
- return field == null ? null : builder.makeInputRef(field.getType(),
field.getIndex());
-}
-List paths = Lists.newArrayList();
-while (rootSegment != null) {
- paths.add(rootSegment.getPath());
- rootSegment = (PathSegment.NameSegment) rootSegment.getChild();
+PathSegment pathSegment = path.getRootSegment();
+
+RelDataTypeField field = findField(pathSegment.getNameSegment().getPath(),
newRowType);
+RexNode rexNode = field == null ? null :
builder.makeInputRef(field.getType(), field.getIndex());
+while (!pathSegment.isLastPath()) {
+ pathSegment = pathSegment.getChild();
+ RexNode ref;
+ if (pathSegment.isNamed()) {
+ref = builder.makeLiteral(pathSegment.getNameSegment().getPath());
+ } else {
+ref =
builder.makeBigintLiteral(BigDecimal.valueOf(pathSegment.getArraySegment().getIndex()));
Review comment:
Out of curiosity, what is BigDecimal doing here? Why not just an
integer?
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java
##
@@ -56,7 +59,23 @@
AbstractWriter getWriter(PhysicalOperator child, String location,
List partitionColumns) throws IOException;
- Set getOptimizerRules();
+ @Deprecated
+ default Set getOptimizerRules() {
+return Collections.emptySet();
+ }
+
+ default Set getOptimizerRules(PlannerPhase phase) {
+switch (phase) {
+ case PHYSICAL:
+return getOptimizerRules();
+ case LOGICAL:
+ case JOIN_PLANNING:case LOGICAL_PRUNE_AND_JOIN:
Review comment:
Is this meant to be on one line?
##
File path: contrib/format-iceberg/README.md
##
@@ -0,0 +1,117 @@
+# Apache Iceberg format plugin
+
+This format plugin enabled Drill to query Apache Iceberg tables.
+
+Unlike regular format plugins, the Iceberg table is a folder with data and
metadata files, but Drill checks the presence
+of the `metadata` folder to ensure that the table is Iceberg one.
+
+Drill supports reading all formats of Iceberg tables available at this moment:
Parquet, Avro, and ORC.
+No need to provide actual table format, it will be discovered automatically.
+
+For details related to Apache Iceberg table format, please refer to [official
docs](https://iceberg.apache.org/#).
+
+## Supported optimizations and features
+
+### Project pushdown
+
+This format plugin supports project and filter pushdown optimizations.
+
+For the case of project pushdown, only specified in the query columns will be
read, even if it is a nested column. In
Review comment:
```suggestion
For the case of project pushdown, only columns specified in the query will
be read, even they are nested columns. In
```
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/IcebergWork.java
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
[jira] [Commented] (DRILL-8042) Select star from MongoDB with aggregated pipeline fails with empty $project error
[ https://issues.apache.org/jira/browse/DRILL-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442438#comment-17442438 ] ASF GitHub Bot commented on DRILL-8042: --- vvysotskyi opened a new pull request #2369: URL: https://github.com/apache/drill/pull/2369 # [DRILL-8042](https://issues.apache.org/jira/browse/DRILL-8042): Select star from MongoDB with aggregated pipeline fails with empty $project error ## Description Updated code to prevent creating aggregates project for empty pipeline and updated mongo plugin implementor to prevent project pushdown for the case of star queries. ## Documentation NA ## Testing Added unit test. closes #2367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Select star from MongoDB with aggregated pipeline fails with empty $project > error > - > > Key: DRILL-8042 > URL: https://issues.apache.org/jira/browse/DRILL-8042 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.0 > > > See https://github.com/apache/drill/issues/2367 for details -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (DRILL-8042) Select star from MongoDB with aggregated pipeline fails with empty $project error
Vova Vysotskyi created DRILL-8042: - Summary: Select star from MongoDB with aggregated pipeline fails with empty $project error Key: DRILL-8042 URL: https://issues.apache.org/jira/browse/DRILL-8042 Project: Apache Drill Issue Type: Bug Affects Versions: 1.20.0 Reporter: Vova Vysotskyi Assignee: Vova Vysotskyi Fix For: 1.20.0 See https://github.com/apache/drill/issues/2367 for details -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8041) Fix mongo scan spec deserialization
[ https://issues.apache.org/jira/browse/DRILL-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442417#comment-17442417 ] ASF GitHub Bot commented on DRILL-8041: --- vvysotskyi commented on pull request #2368: URL: https://github.com/apache/drill/pull/2368#issuecomment-966567449 @cgivre, no, it is a separate issue, I'm working on it right now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix mongo scan spec deserialization > --- > > Key: DRILL-8041 > URL: https://issues.apache.org/jira/browse/DRILL-8041 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.0 > > > See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8041) Fix mongo scan spec deserialization
[ https://issues.apache.org/jira/browse/DRILL-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442416#comment-17442416 ] ASF GitHub Bot commented on DRILL-8041: --- cgivre commented on pull request #2368: URL: https://github.com/apache/drill/pull/2368#issuecomment-966566791 @vvysotskyi Does this PR also address this issue: https://github.com/apache/drill/issues/2367? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix mongo scan spec deserialization > --- > > Key: DRILL-8041 > URL: https://issues.apache.org/jira/browse/DRILL-8041 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.0 > > > See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8041) Fix mongo scan spec deserialization
[ https://issues.apache.org/jira/browse/DRILL-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442407#comment-17442407 ] ASF GitHub Bot commented on DRILL-8041: --- vvysotskyi opened a new pull request #2368: URL: https://github.com/apache/drill/pull/2368 # [DRILL-8041](https://issues.apache.org/jira/browse/DRILL-8041): Fix mongo scan spec deserialization ## Description Fixed deserialization of mongo scan spec classes. ## Documentation NA ## Testing Added UT closes #2355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix mongo scan spec deserialization > --- > > Key: DRILL-8041 > URL: https://issues.apache.org/jira/browse/DRILL-8041 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.0 > > > See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (DRILL-8041) Fix mongo scan spec deserialization
[ https://issues.apache.org/jira/browse/DRILL-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Vysotskyi updated DRILL-8041: -- Fix Version/s: 1.20.0 > Fix mongo scan spec deserialization > --- > > Key: DRILL-8041 > URL: https://issues.apache.org/jira/browse/DRILL-8041 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.0 > > > See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (DRILL-8041) Fix mongo scan spec deserialization
[ https://issues.apache.org/jira/browse/DRILL-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Vysotskyi updated DRILL-8041: -- Affects Version/s: 1.20.0 > Fix mongo scan spec deserialization > --- > > Key: DRILL-8041 > URL: https://issues.apache.org/jira/browse/DRILL-8041 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > > See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (DRILL-8041) Fix mongo scan spec deserialization
Vova Vysotskyi created DRILL-8041: - Summary: Fix mongo scan spec deserialization Key: DRILL-8041 URL: https://issues.apache.org/jira/browse/DRILL-8041 Project: Apache Drill Issue Type: Bug Reporter: Vova Vysotskyi Assignee: Vova Vysotskyi See https://github.com/apache/drill/issues/2355 for details. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442377#comment-17442377
]
ASF GitHub Bot commented on DRILL-8027:
---
vvysotskyi commented on a change in pull request #2357:
URL: https://github.com/apache/drill/pull/2357#discussion_r747693599
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/format/IcebergFormatPlugin.java
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.iceberg.format;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.metastore.MetadataProviderManager;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.AbstractWriter;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.PlannerPhase;
+import org.apache.drill.exec.planner.common.DrillStatsTable;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.PluginRulesProviderImpl;
+import org.apache.drill.exec.store.StoragePluginRulesSupplier;
+import org.apache.drill.exec.store.dfs.FileSelection;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FormatMatcher;
+import org.apache.drill.exec.store.dfs.FormatPlugin;
+import org.apache.drill.exec.store.iceberg.IcebergGroupScan;
+import org.apache.drill.exec.store.iceberg.plan.IcebergPluginImplementor;
+import org.apache.drill.exec.store.plan.rel.PluginRel;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+public class IcebergFormatPlugin implements FormatPlugin {
+
+ private static final String ICEBERG_CONVENTION_PREFIX = "ICEBERG.";
+
+ private final FileSystemConfig storageConfig;
+
+ private final IcebergFormatPluginConfig config;
+
+ private final Configuration fsConf;
+
+ private final DrillbitContext context;
+
+ private final String name;
+
+ private final IcebergFormatMatcher matcher;
+
+ private final StoragePluginRulesSupplier storagePluginRulesSupplier;
+
+ public IcebergFormatPlugin(
+String name,
+DrillbitContext context,
+Configuration fsConf,
+FileSystemConfig storageConfig,
+IcebergFormatPluginConfig config) {
+this.storageConfig = storageConfig;
+this.config = config;
+this.fsConf = fsConf;
+this.context = context;
+this.name = name;
+this.matcher = new IcebergFormatMatcher(this);
+this.storagePluginRulesSupplier = storagePluginRulesSupplier(name);
+ }
+
+ private static StoragePluginRulesSupplier storagePluginRulesSupplier(String
name) {
+Convention convention = new Convention.Impl(ICEBERG_CONVENTION_PREFIX +
name, PluginRel.class);
+return StoragePluginRulesSupplier.builder()
+ .rulesProvider(new PluginRulesProviderImpl(convention,
IcebergPluginImplementor::new))
+ .supportsFilterPushdown(true)
+ .supportsProjectPushdown(true)
+ .convention(convention)
+ .build();
+ }
+
+ @Override
+ public boolean supportsRead() {
+return true;
+ }
+
+ @Override
+ public boolean supportsWrite() {
Review comment:
Yep, in the future, we definitely should add it.
##
File path:
contrib/format-iceberg/src/main/java/org/apache/drill/exec/store/iceberg/IcebergGroupScan.java
##
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the
[jira] [Commented] (DRILL-7983) Add a REST API to support the get running or completed profiles
[
https://issues.apache.org/jira/browse/DRILL-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442266#comment-17442266
]
ASF GitHub Bot commented on DRILL-7983:
---
cgivre commented on pull request #2353:
URL: https://github.com/apache/drill/pull/2353#issuecomment-966278842
@CuteKittyhoho
Did you see the review comments from Paul? I also think this would be ok to
merge, but some additional unit tests would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a REST API to support the get running or completed profiles
> ---
>
> Key: DRILL-7983
> URL: https://issues.apache.org/jira/browse/DRILL-7983
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Cong Luo
>Priority: Major
> Fix For: 1.20.0
>
>
> Drill have a REST API to get the profiles of running and completed queries.
> this goal of ticket is extended from the `/profiles.json` to support get the
> running or completed profiles (not all of them). the client does not need to
> filter with the state.
> old URI
> {code:java}
> /profiles.json{code}
> new URI
> {code:java}
> /profiles.json
> /profiles/running
> /profiles/completed{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-7983) Add a REST API to support the get running or completed profiles
[
https://issues.apache.org/jira/browse/DRILL-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442268#comment-17442268
]
ASF GitHub Bot commented on DRILL-7983:
---
cgivre edited a comment on pull request #2353:
URL: https://github.com/apache/drill/pull/2353#issuecomment-966278842
@CuteKittyhoho
Did you see the review comments from Paul? I also think this would be ok to
merge, but some additional unit tests would be nice.
Also, can you please rebase on current master and resolve merge conflicts?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a REST API to support the get running or completed profiles
> ---
>
> Key: DRILL-7983
> URL: https://issues.apache.org/jira/browse/DRILL-7983
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Cong Luo
>Priority: Major
> Fix For: 1.20.0
>
>
> Drill have a REST API to get the profiles of running and completed queries.
> this goal of ticket is extended from the `/profiles.json` to support get the
> running or completed profiles (not all of them). the client does not need to
> filter with the state.
> old URI
> {code:java}
> /profiles.json{code}
> new URI
> {code:java}
> /profiles.json
> /profiles/running
> /profiles/completed{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8027) Format plugin for Apache Iceberg
[ https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442197#comment-17442197 ] ASF GitHub Bot commented on DRILL-8027: --- Z0ltrix commented on a change in pull request #2357: URL: https://github.com/apache/drill/pull/2357#discussion_r747358905 ## File path: pom.xml ## @@ -137,6 +137,7 @@ 1.18.20 0.1.1 0.20 +0.12.0 Review comment: theres a new iceberg release https://github.com/apache/iceberg/releases/tag/apache-iceberg-0.12.1 Maybe we should use it directly? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Format plugin for Apache Iceberg > > > Key: DRILL-8027 > URL: https://issues.apache.org/jira/browse/DRILL-8027 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Labels: plugin > Fix For: 1.20.0 > > > Implement a format plugin for Apache Iceberg. > Plugin should be able to: > - support reading data from Iceberg tables in Parquet, Avro, and ORC formats > - push down fields used in the project > - push down supported filter expressions > - spit and parallelize reading tasks > - provide a way for specifying Iceberg-specific configurations > - read specific snapshot versions if configured > - read table metadata (entries, files, history, snapshots, manifests, > partitions, etc.) > - support schema provisioning -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (DRILL-8019) Use HTTP error codes to signal failures in the REST API
[ https://issues.apache.org/jira/browse/DRILL-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8019. --- Resolution: Duplicate > Use HTTP error codes to signal failures in the REST API > --- > > Key: DRILL-8019 > URL: https://issues.apache.org/jira/browse/DRILL-8019 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.19.0 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.20.0 > > > Unfortunately the Drill web UI relies on errors being communicated using HTTP > 200 and message saying something other than "Success" so it needs updating at > the same time. -- This message was sent by Atlassian Jira (v8.20.1#820001)
