[jira] [Created] (DRILL-7713) Upgrade misc libraries which outdated versions have reported vulnerabilities
Arina Ielchiieva created DRILL-7713: --- Summary: Upgrade misc libraries which outdated versions have reported vulnerabilities Key: DRILL-7713 URL: https://issues.apache.org/jira/browse/DRILL-7713 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 List of libraries to update: |commons-beanutils-1.9.2.jar| |jackson-databind-2.9.9.jar| |xalan-2.7.1.jar| |commons-compress-1.18.jar| |metadata-extractor-2.11.0.jar| |xercesImpl-2.11.0.jar| |retrofit-2.1.0.jar| |snakeyaml-1.23.jar| |commons-codec-1.10.jar| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7712) Fix issues after ZK upgrade
Arina Ielchiieva created DRILL-7712:
---
Summary: Fix issues after ZK upgrade
Key: DRILL-7712
URL: https://issues.apache.org/jira/browse/DRILL-7712
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.18.0
Reporter: Arina Ielchiieva
Assignee: Vova Vysotskyi
Fix For: 1.18.0
Warnings during jdbc-all build (absent when building with Mapr profile):
{noformat}
netty-transport-native-epoll-4.1.45.Final.jar,
netty-transport-native-epoll-4.0.48.Final-linux-x86_64.jar define 46
overlapping classes:
- io.netty.channel.epoll.AbstractEpollStreamChannel$2
- io.netty.channel.epoll.AbstractEpollServerChannel$EpollServerSocketUnsafe
- io.netty.channel.epoll.EpollDatagramChannel
- io.netty.channel.epoll.AbstractEpollStreamChannel$SpliceInChannelTask
- io.netty.channel.epoll.NativeDatagramPacketArray
- io.netty.channel.epoll.EpollSocketChannelConfig
- io.netty.channel.epoll.EpollTcpInfo
- io.netty.channel.epoll.EpollEventArray
- io.netty.channel.epoll.EpollEventLoop
- io.netty.channel.epoll.EpollSocketChannel
- 36 more...
netty-transport-native-unix-common-4.1.45.Final.jar,
netty-transport-native-epoll-4.0.48.Final-linux-x86_64.jar define 15
overlapping classes:
- io.netty.channel.unix.Errors$NativeConnectException
- io.netty.channel.unix.ServerDomainSocketChannel
- io.netty.channel.unix.DomainSocketAddress
- io.netty.channel.unix.Socket
- io.netty.channel.unix.NativeInetAddress
- io.netty.channel.unix.DomainSocketChannelConfig
- io.netty.channel.unix.Errors$NativeIoException
- io.netty.channel.unix.DomainSocketReadMode
- io.netty.channel.unix.ErrorsStaticallyReferencedJniMethods
- io.netty.channel.unix.UnixChannel
- 5 more...
maven-shade-plugin has detected that some class files are
present in two or more JARs. When this happens, only one
single version of the class is copied to the uber jar.
Usually this is not harmful and you can skip these warnings,
otherwise try to manually exclude artifacts based on
mvn dependency:tree -Ddetail=true and the above output.
See http://maven.apache.org/plugins/maven-shade-plugin/
{noformat}
Additional warning build with Mapr profile:
{noformat}
The following patterns were never triggered in this artifact inclusion filter:
o 'org.apache.zookeeper:zookeeper-jute'
{noformat}
NPEs in tests (though tests do not fail):
{noformat}
[INFO] Running org.apache.drill.exec.coord.zk.TestZookeeperClient
4880
java.lang.NullPointerException
4881
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:269)
4882
at
org.apache.zookeeper.server.ZKDatabase.fastForwardDataBase(ZKDatabase.java:251)
4883
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:583)
4884
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:546)
4885
at
org.apache.zookeeper.server.NIOServerCnxnFactory.shutdown(NIOServerCnxnFactory.java:
{noformat}
{noformat}
[INFO] Running org.apache.drill.exec.coord.zk.TestEphemeralStore
5278
java.lang.NullPointerException
5279
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:269)
5280
at org.apache.zookeepe
{noformat}
{noformat}
[INFO] Running org.apache.drill.yarn.zk.TestAmRegistration
6767
java.lang.NullPointerException
6768
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:269)
6769
at
org.apache.zookeeper.server.ZKDatabase.fastForwardDataBase(ZKDatabase.java:251)
6770
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:583)
6771
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:546)
6772
at
org.apache.zookeeper.server.NIOServerCnxnFactory.shutdown(NIOServerCnxnFactory.java:929)
6773
at org.apache.curator.t
{noformat}
{noformat}
org.apache.drill.yarn.client.TestCommandLineOptions
6823
java.lang.NullPointerException
6824
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:269)
6825
at
org.apache.zookeeper.server.ZKDatabase.fastForwardDataBase(ZKDatabase.java:251)
6826
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:583)
6827
at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:546)
6828
at org.apac
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7710) Fix TestMetastoreCommands#testDefaultSegment test
Arina Ielchiieva created DRILL-7710:
---
Summary: Fix TestMetastoreCommands#testDefaultSegment test
Key: DRILL-7710
URL: https://issues.apache.org/jira/browse/DRILL-7710
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Assignee: Vova Vysotskyi
Fix For: 1.18.0
Test {{TestMetastoreCommands#testDefaultSegment}} sometimes fails:
{noformat}
[ERROR] TestMetastoreCommands.testDefaultSegment:1870
expected: but was:
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7707) Unable to analyze table metadata is it resides in non-writable workspace
Arina Ielchiieva created DRILL-7707:
---
Summary: Unable to analyze table metadata is it resides in
non-writable workspace
Key: DRILL-7707
URL: https://issues.apache.org/jira/browse/DRILL-7707
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Unable to analyze table metadata is it resides in non-writable workspace:
{noformat}
apache drill> analyze table cp.`employee.json` refresh metadata;
Error: VALIDATION ERROR: Unable to create or drop objects. Schema [cp] is
immutable.
{noformat}
Stacktrace:
{noformat}
[Error Id: b7f233cd-f090-491e-a487-5fc4c25444a4 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657)
at
org.apache.drill.exec.planner.sql.SchemaUtilites.resolveToDrillSchemaInternal(SchemaUtilites.java:230)
at
org.apache.drill.exec.planner.sql.SchemaUtilites.resolveToDrillSchema(SchemaUtilites.java:208)
at
org.apache.drill.exec.planner.sql.handlers.DrillTableInfo.getTableInfoHolder(DrillTableInfo.java:101)
at
org.apache.drill.exec.planner.sql.handlers.MetastoreAnalyzeTableHandler.getPlan(MetastoreAnalyzeTableHandler.java:108)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7706) Drill RDBMS Metastore
Arina Ielchiieva created DRILL-7706: --- Summary: Drill RDBMS Metastore Key: DRILL-7706 URL: https://issues.apache.org/jira/browse/DRILL-7706 Project: Apache Drill Issue Type: New Feature Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 Currently Drill has only one Metastore implementation based on Iceberg tables. Iceberg tables are file based storage that supports concurrent writes / reads but required to be placed on distributed file system. This Jira aims to implement Drill RDBMS Metastore which will store Drill Metastore metadata in the database of the user's choice. Currently, PostgreSQL and MySQL databases are supported, others might work as well but no testing was done. Also out of box for demonstration / testing purposes Drill will setup SQLite file based embedded database but this is only applicable for Drill in embedded mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7704) Update Maven dependency
Arina Ielchiieva created DRILL-7704: --- Summary: Update Maven dependency Key: DRILL-7704 URL: https://issues.apache.org/jira/browse/DRILL-7704 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 Currently, minimal Maven version in Drill is 3.3.3, it's old contains dependency to the plexus-utils-3.0.20 library which has reported vulnerabilities. This Jira aims to update Maven version to 3.6.3. Having latest maven version is also crucial when using maven plugins that depend on the latest version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7528) Update Avro format plugin documentation
[ https://issues.apache.org/jira/browse/DRILL-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-7528. - Resolution: Fixed Added in https://github.com/apache/drill/commit/0e17eea19aca27b88c98778fcfb7057a45501ab9. > Update Avro format plugin documentation > --- > > Key: DRILL-7528 > URL: https://issues.apache.org/jira/browse/DRILL-7528 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.18.0 > > > Currently documentation states that Avro plugin is experimental. > As per Drill 1.17 / 1.18 it's code is pretty stable (since Drill 1.18 it uses > EVF). > Documentation should be updated accordingly. > https://drill.apache.org/docs/querying-avro-files/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7672) Make metadata type required when reading from / writing into Drill Metastore
Arina Ielchiieva created DRILL-7672: --- Summary: Make metadata type required when reading from / writing into Drill Metastore Key: DRILL-7672 URL: https://issues.apache.org/jira/browse/DRILL-7672 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 Metastore consists of components: TABLES, VIEWS etc (so far only TABLES are implemented). Each component metadata can have types. For examples, TABLES metadata can be of the following types: TABLE, SEGMENT, FILE, ROW_GROUP, PARTITION. During initial Metastore implementation when reading from / writing into Metastore, metadata type was indicated in filter expressions. For Iceberg Metastore where all data is stored in files this was not this critical, basically when information is retrieved about the table, table folder is queried. For other Metastore implementations knowing metadata type can be more critical. For example, RDBMS Metastore would store TABLES metadata in different tables thus knowing which table to query would improve performance rather than trying to query all tables. Of course, we could traverse query filter and look for the hints which metadata type is needed but it is much better to know required metadata type beforehand without any extra logic. Taking into account that Metastore metadata is queried only in Drill code, developer knows beforehand what he needs to get / update / delete. This Jira aims to make metadata type required when reading from / writing into Drill Metastore. This change does not have any affect on the users, just internal code refactoring. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7665) Add UNION to schema parser
Arina Ielchiieva created DRILL-7665: --- Summary: Add UNION to schema parser Key: DRILL-7665 URL: https://issues.apache.org/jira/browse/DRILL-7665 Project: Apache Drill Issue Type: Improvement Reporter: Arina Ielchiieva Fix For: 1.18.0 After DRILL-7633 has defined proper type string for UNION it should be added to schema parser to allow proper ser / de. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7624) When Hive plugin is enabled with default config, cannot execute any SQL query
[
https://issues.apache.org/jira/browse/DRILL-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-7624.
-
Resolution: Fixed
Merged with commit id 63e64c2a8156a66ba1b1c8c1ec62e8da467bbbc9.
> When Hive plugin is enabled with default config, cannot execute any SQL query
> -
>
> Key: DRILL-7624
> URL: https://issues.apache.org/jira/browse/DRILL-7624
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: Dmytro Kondriukov
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.18.0
>
>
> *Preconditions:*
> Enable "hive" plugin, without editing configuration (default config)
> *Steps:*
> Run any valid query:
> {code:sql}
> SELECT 100;
> {code}
> *Expected result:* The query should be successfully executed.
> *Actual result:* "UserRemoteException : INTERNAL_ERROR ERROR: Failure
> setting up Hive metastore client."
> {noformat}
> org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR:
> Failure setting up Hive metastore client.
> Plugin name hive
> Plugin class org.apache.drill.exec.store.hive.HiveStoragePlugin
> Please, refer to logs for more information.
> [Error Id: db44f5c3-5136-4fc6-8158-50b63d775fe0 ]
> {noformat}
>
> {noformat}
> (org.apache.drill.common.exceptions.ExecutionSetupException) Failure
> setting up Hive metastore client.
> org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.():78
> org.apache.drill.exec.store.hive.HiveStoragePlugin.():77
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():62
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():423
> org.apache.drill.exec.store.ClassicConnectorLocator.create():274
> org.apache.drill.exec.store.ConnectorHandle.newInstance():98
> org.apache.drill.exec.store.PluginHandle.plugin():143
>
> org.apache.drill.exec.store.StoragePluginRegistryImpl$PluginIterator.next():616
>
> org.apache.drill.exec.store.StoragePluginRegistryImpl$PluginIterator.next():601
> org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig.getRules():48
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():367
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():351
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():338
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel():663
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():198
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():169
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():283
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():163
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan():140
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():93
> org.apache.drill.exec.work.foreman.Foreman.runSQL():590
> org.apache.drill.exec.work.foreman.Foreman.run():275
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (org.apache.hadoop.hive.metastore.api.MetaException) Unable to
> open a test connection to the given database. JDBC url =
> jdbc:derby:;databaseName=../sample-data/drill_hive_db;create=true, username =
> APP. Terminating connection pool (set lazyInit to true if you expect to start
> your database after your app). Original Exception: --
> java.sql.SQLException: Failed to create database
> '../sample-data/drill_hive_db', see the next exception for details.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection40.(Unknown Source)
> at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
> at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
> at org.apache.derby.jdbc.Driver20.connect(Unknown Source)
> at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at
[jira] [Created] (DRILL-7639) Replace DBCP2 with HikariCP in RDBMS (JDBC) plugin
Arina Ielchiieva created DRILL-7639: --- Summary: Replace DBCP2 with HikariCP in RDBMS (JDBC) plugin Key: DRILL-7639 URL: https://issues.apache.org/jira/browse/DRILL-7639 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Hikari is much faster and reliable than DBCP2. See comparison and benchmarks: https://beansroasted.wordpress.com/2017/07/29/connection-pool-analysis/ https://github.com/brettwooldridge/HikariCP -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7627) Update MySql version in JdbcStoragePlugin tests and cache ~/.embedmysql
Arina Ielchiieva created DRILL-7627:
---
Summary: Update MySql version in JdbcStoragePlugin tests and cache
~/.embedmysql
Key: DRILL-7627
URL: https://issues.apache.org/jira/browse/DRILL-7627
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.18.0
When running tests on clean env (or when ~/.embedmysql is cleaned), currently
there are issues when downloading MySQL version v5_6_21, download hangs and
JdbcStoragePlugin tests fail with timeout.
{noformat}
Download Version 5.6.21:Linux:B64 START
Download Version 5.6.21:Linux:B64 DownloadSize: 311516309
Download Version 5.6.21:Linux:B64 0%
Download Version 5.6.21:Linux:B64 5%
Download Version 5.6.21:Linux:B64 10%
Download Version 5.6.21:Linux:B64 15%
Download Version 5.6.21:Linux:B64 20%
Download Version 5.6.21:Linux:B64 25%
Download Version 5.6.21:Linux:B64 30%
Download Version 5.6.21:Linux:B64 35%
Download Version 5.6.21:Linux:B64 40%
Download Version 5.6.21:Linux:B64 45%
Download Version 5.6.21:Linux:B64 50%
Download Version 5.6.21:Linux:B64 55%
Download Version 5.6.21:Linux:B64 60%
TestJdbcPluginWithMySQLIT.initMysql:70 » Distribution java.net.SocketTimeoutEx.
{noformat}
Workaround is manually to download MySQL:
{noformat}
mkdir -p ~/.embedmysql/MySQL-5.6
wget -P ~/Downloads
http://mirror.cogentco.com/pub/mysql/MySQL-5.6/mysql-5.6.21-linux-glibc2.5-x86_64.tar.gz
cp ~/Downloads/mysql-5.6.21-linux-glibc2.5-x86_64.tar.gz
~/.embedmysql/MySQL-5.6/mysql-5.6.21-linux-glibc2.5-x86_64.tar.gz
{noformat}
Upgrading to the latest available MySQL version (5_7_19) fixes this issue.
Also it would be nice to cache ~/.embedmysql folder during GitHub Actions CI
run to spare time spent on downloading and unpacking.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7617) Disabled storage plugins configuration is not displayed on Web UI
Arina Ielchiieva created DRILL-7617:
---
Summary: Disabled storage plugins configuration is not displayed
on Web UI
Key: DRILL-7617
URL: https://issues.apache.org/jira/browse/DRILL-7617
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.18.0
Reporter: Arina Ielchiieva
Assignee: Paul Rogers
Fix For: 1.18.0
After DRILL-7590, disabled storage plugins are displayed on Web UI but if you
press update button their content is not shown, {{null} is shown instead. If
you check plugin file it does contain configuration information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7614) Add helpful tips to Testing.md
Arina Ielchiieva created DRILL-7614: --- Summary: Add helpful tips to Testing.md Key: DRILL-7614 URL: https://issues.apache.org/jira/browse/DRILL-7614 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Fix For: 1.18.0 Add some helpful tips to Testing.md, see PR for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7459) Fetch size does not work on Postgres JDBC plugin
[
https://issues.apache.org/jira/browse/DRILL-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-7459.
-
Resolution: Fixed
Was fixed in the scope of DRILL-7467.
The following plugin configuration should be used to support batches:
{noformat}
{
"type": "jdbc",
"driver": "org.postgresql.Driver",
"url": "jdbc:postgresql://localhost:5959/my_db?defaultRowFetchSize=2",
"username": "my_user",
"password": "my_pass",
"caseInsensitiveTableNames": false,
"sourceParameters": {
"defaultAutoCommit": false
},
"enabled": true
}
{noformat}
> Fetch size does not work on Postgres JDBC plugin
>
>
> Key: DRILL-7459
> URL: https://issues.apache.org/jira/browse/DRILL-7459
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.15.0
>Reporter: Priyanka Bhoir
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.18.0
>
>
> To prevent the drill from going out of memory, it is suggested to set the
> fetch size in Jdbc URL to enable data streaming([#DRILL-6794 |
> https://issues.apache.org/jira/browse/DRILL-6794] discusses this). This does
> not work on Postgres for the following reason:
> For fetchSize size to work on Postgres, the connection must not be in
> autocommit mode. There is no parameter to set autocommit to false in
> connection string other than programmatically calling
> conn.setAutoCommit(false).
> See
> [https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example]
> See [https://jdbc.postgresql.org/documentation/head/connect.html] for the
> list of all connection string properties.
> Fix is to add a property 'defaultAutoCommit' to JdbcStorageConfig and call
> BasicDataSource#setDefaultAutoCommit in JdbcStoragePlugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7549) Fix validation error when querying absent folder in embedded mode
Arina Ielchiieva created DRILL-7549:
---
Summary: Fix validation error when querying absent folder in
embedded mode
Key: DRILL-7549
URL: https://issues.apache.org/jira/browse/DRILL-7549
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.18.0
{noformat}
apache drill> select * from dfs.tmp.`abc.parquet`;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
'abc.parquet' not found within 'dfs.tmp'
[Error Id: 0dad391e-ea4d-4d13-95e7-218dec865ad2 ] (state=,code=0)
apache drill> select * from dfs.tmp.`abc/abc`;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
'abc/abc' not found within 'dfs.tmp'
[Error Id: 94ea83c7-4983-4958-8a82-eea49b58d4ed ] (state=,code=0)
apache drill> use dfs.tmp;
+--+-+
| ok | summary |
+--+-+
| true | Default schema changed to [dfs.tmp] |
+--+-+
1 row selected (0.277 seconds)
apache drill (dfs.tmp)> select * from `abc.parquet`;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 27: Object
'abc.parquet' not found
[Error Id: d814df53-2a94-46da-81e5-f49a3d50a665 ] (state=,code=0)
apache drill (dfs.tmp)> select * from `abc/abc`;
Error: VALIDATION ERROR: null
[Error Id: 6c11d397-f893-4ef6-9832-4a96a2029f7d ] (state=,code=0)
{noformat}
Full error:
{noformat}
Caused by: java.lang.IllegalArgumentException:
at
org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkArgument(Preconditions.java:121)
at
org.apache.drill.exec.store.sys.store.LocalPersistentStore.makePath(LocalPersistentStore.java:147)
at
org.apache.drill.exec.store.sys.store.LocalPersistentStore.get(LocalPersistentStore.java:166)
at
org.apache.drill.exec.store.sys.CaseInsensitivePersistentStore.get(CaseInsensitivePersistentStore.java:42)
at
org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:167)
at
org.apache.calcite.jdbc.DynamicRootSchema.loadSchemaFactory(DynamicRootSchema.java:80)
at
org.apache.calcite.jdbc.DynamicRootSchema.getImplicitSubSchema(DynamicRootSchema.java:67)
at
org.apache.calcite.jdbc.CalciteSchema.getSubSchema(CalciteSchema.java:265)
at
org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:133)
at
org.apache.calcite.sql.validate.EmptyScope.resolveTable(EmptyScope.java:99)
at
org.apache.calcite.sql.validate.DelegatingScope.resolveTable(DelegatingScope.java:203)
at
org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:105)
at
org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:177)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3129)
at
org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3111)
at
org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3383)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:944)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:651)
at
org.apache.drill.exec.planner.sql.conversion.SqlConverter.validate(SqlConverter.java:189)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:648)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:170)
[jira] [Resolved] (DRILL-7466) Configurable number of connection in JDBC storage connection pool
[ https://issues.apache.org/jira/browse/DRILL-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-7466. - Resolution: Fixed Merged into master with commit id 50cd931da0364355535029db8e9d7a1445218803. > Configurable number of connection in JDBC storage connection pool > - > > Key: DRILL-7466 > URL: https://issues.apache.org/jira/browse/DRILL-7466 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - JDBC >Affects Versions: 1.17.0 >Reporter: Priyanka Bhoir >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.18.0 > > > Drill does not allow the number of connections in the pool to configured > When a JDBC storage plugin is created, it creates 8 connections per plugin > which happen to be the default for dbcp connection pool. Currently, there is > no way to configure these parameters using storage configuration. Max Idle is > set to 8 leaving all connections open even when unused. This situation > creates an unnecessary connection to DB. > These parameters must be made configuration in the JDBC storage plugin > configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-6738) validationQuery option for JDBC storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-6738. - Resolution: Fixed Merged into master with commit id 50cd931da0364355535029db8e9d7a1445218803. > validationQuery option for JDBC storage plugin > -- > > Key: DRILL-6738 > URL: https://issues.apache.org/jira/browse/DRILL-6738 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - JDBC >Affects Versions: 1.14.0 > Environment: Apache Drill version 1.14.0 running on CentOS 7.0.1406. > MySQL version 5.5.43 running on CentOS 6.4. > MySQL connector/j version 5.1.44. >Reporter: Cheolgoo Kang >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.18.0 > > > Currently JDBC storage plugin uses DBCP2 for the connection pooling, but it > wouldn't validate the connection's still available or not, and it'd just spit > the connection lost error. > We should be able to have a validationQuery option described > [here|https://commons.apache.org/proper/commons-dbcp/configuration.html] in > Drill's JDBC storage plugin configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-7398) dfs avro file support to treat non existent fields as null
[ https://issues.apache.org/jira/browse/DRILL-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-7398. - Resolution: Fixed Merged into master with commit id bf7277c9c8725d6b9a56988f72c31ede1d486b85. > dfs avro file support to treat non existent fields as null > -- > > Key: DRILL-7398 > URL: https://issues.apache.org/jira/browse/DRILL-7398 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Avro >Reporter: Dan Schmitt >Priority: Major > > It would be nice to have an option to return null for non existing fields. > Currently we have a use case of evolving avro file schemas where things get > added. > It would be nice to be able to query old and new data without having to > rewrite the old data with a new schema with null/optional/default values. > A short term fix would be to provide an option to return null with dfs for > avro fields that aren't found in a file. > A nicer fix would be to support a resolving schema that can be given to dfs > to read the avro files to present a uniform interface (so aliases, defaults, > etc. could be utilized without rewriting each avro file.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7544) Upgrade Iceberg version to support Parquet 1.11.0
Arina Ielchiieva created DRILL-7544: --- Summary: Upgrade Iceberg version to support Parquet 1.11.0 Key: DRILL-7544 URL: https://issues.apache.org/jira/browse/DRILL-7544 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 Upgrade Iceberg versions and load it using JitPack after Iceberg will support Parquet 1.11 (https://github.com/apache/incubator-iceberg/pull/708). Also remove workaround in ExpirationHandler since https://github.com/apache/incubator-iceberg/issues/181 is resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7542) Fix Drill-on-Yarn logger
Arina Ielchiieva created DRILL-7542:
---
Summary: Fix Drill-on-Yarn logger
Key: DRILL-7542
URL: https://issues.apache.org/jira/browse/DRILL-7542
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0, 1.16.0
Reporter: Arina Ielchiieva
Drill project uses Logback logger backed by SLF4J:
{noformat}
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
private static final Logger logger =
LoggerFactory.getLogger(ResultsListener.class);
{noformat}
Drill-on-Yarn project uses commons loggin:
{noformat}
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
private static final Log LOG = LogFactory.getLog(AbstractScheduler.class);
{noformat}
It would be nice if all project components used the same approach for logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7537) Convert Parquet format to EVF
Arina Ielchiieva created DRILL-7537: --- Summary: Convert Parquet format to EVF Key: DRILL-7537 URL: https://issues.apache.org/jira/browse/DRILL-7537 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7535) Convert Lstv to EVF
Arina Ielchiieva created DRILL-7535: --- Summary: Convert Lstv to EVF Key: DRILL-7535 URL: https://issues.apache.org/jira/browse/DRILL-7535 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7538) Convert MaprDB format to EVF
Arina Ielchiieva created DRILL-7538: --- Summary: Convert MaprDB format to EVF Key: DRILL-7538 URL: https://issues.apache.org/jira/browse/DRILL-7538 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7536) Convert Image format to EVF
Arina Ielchiieva created DRILL-7536: --- Summary: Convert Image format to EVF Key: DRILL-7536 URL: https://issues.apache.org/jira/browse/DRILL-7536 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7534) Convert Httpd to EVF
Arina Ielchiieva created DRILL-7534: --- Summary: Convert Httpd to EVF Key: DRILL-7534 URL: https://issues.apache.org/jira/browse/DRILL-7534 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Charles Givre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7533) Convert Pcapng to EVF
Arina Ielchiieva created DRILL-7533: --- Summary: Convert Pcapng to EVF Key: DRILL-7533 URL: https://issues.apache.org/jira/browse/DRILL-7533 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Charles Givre -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7532) Convert SysLog to EVF
Arina Ielchiieva created DRILL-7532: --- Summary: Convert SysLog to EVF Key: DRILL-7532 URL: https://issues.apache.org/jira/browse/DRILL-7532 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Charles Givre Convert SysLog to EVF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7531) Move format plugins to EVF
Arina Ielchiieva created DRILL-7531: --- Summary: Move format plugins to EVF Key: DRILL-7531 URL: https://issues.apache.org/jira/browse/DRILL-7531 Project: Apache Drill Issue Type: Improvement Reporter: Arina Ielchiieva Fix For: 1.18.0 This is umbrella Jira to track down process of moving format plugins to EVF. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7530) Fix class names in loggers
Arina Ielchiieva created DRILL-7530:
---
Summary: Fix class names in loggers
Key: DRILL-7530
URL: https://issues.apache.org/jira/browse/DRILL-7530
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.18.0
Some loggers have incorrect class names which leads to incorrect information in
logs.
This Jira aims to fix all occurrences of incorrect class names in loggers.
Preliminary list (some occurrences will be excluded):
{noformat}
Name: MapRDBTableCache.java. Expected: MapRDBTableCache. Got:
MapRDBFormatPlugin.
Name: MapRDBTableCache.java. Expected: MapRDBTableCache. Got:
MapRDBFormatPlugin.
Name: HiveFuncHolderExpr.java. Expected: HiveFuncHolderExpr. Got:
DrillFuncHolderExpr.
Name: HiveFuncHolder.java. Expected: HiveFuncHolder. Got:
FunctionImplementationRegistry.
Name: HiveMetadataProvider.java. Expected: HiveMetadataProvider. Got: HiveStats.
Name: TableEntryCacheLoader.java. Expected: TableEntryCacheLoader. Got:
TableNameCacheLoader.
Name: TestKafkaSuit.java. Expected: TestKafkaSuit. Got: LoggerFactory.
Name: DrillTestWrapper.java. Expected: DrillTestWrapper. Got: BaseTestQuery.
Name: TestDisabledFunctionality.java. Expected: TestDisabledFunctionality. Got:
TestExampleQueries.
Name: TestMergeJoin.java. Expected: TestMergeJoin. Got: HashAggBatch.
Name: TestLateralJoinCorrectnessBatchProcessing.java. Expected:
TestLateralJoinCorrectnessBatchProcessing. Got: TestNewLateralJoinCorrectness.
Name: TestOperatorRecordBatch.java. Expected: TestOperatorRecordBatch. Got:
SubOperatorTest.
Name: TestPauseInjection.java. Expected: TestPauseInjection. Got: DummyClass.
Name: TestComplexTypeWriter.java. Expected: TestComplexTypeWriter. Got:
TestComplexTypeReader.
Name: AvgIntervalTypeFunctions.java. Expected: AvgIntervalTypeFunctions. Got:
AvgFunctions.
Name: SSLConfigBuilder.java. Expected: SSLConfigBuilder. Got:
org.apache.drill.exec.ssl.SSLConfigBuilder.
Name: PlannerPhase.java. Expected: PlannerPhase. Got: DrillRuleSets.
Name: AbstractIndexDescriptor.java. Expected: AbstractIndexDescriptor. Got:
AbstractIndexDescriptor .
Name: CoveringPlanNoFilterGenerator.java. Expected:
CoveringPlanNoFilterGenerator. Got: CoveringIndexPlanGenerator.
Name: AbstractSqlSetHandler.java. Expected: AbstractSqlSetHandler. Got:
AbstractSqlHandler.
Name: HashJoinMemoryCalculatorImpl.java. Expected:
HashJoinMemoryCalculatorImpl. Got: BuildSidePartitioning.
Name: HashJoinMemoryCalculatorImpl.java. Expected:
HashJoinMemoryCalculatorImpl. Got: PostBuildCalculationsImpl.
Name: HashJoinMemoryCalculator.java. Expected: HashJoinMemoryCalculator. Got:
PartitionStatSet.
Name: NestedLoopJoinTemplate.java. Expected: NestedLoopJoinTemplate. Got:
NestedLoopJoinBatch.
Name: PartitionLimitRecordBatch.java. Expected: PartitionLimitRecordBatch. Got:
LimitRecordBatch.
Name: HashAggTemplate.java. Expected: HashAggTemplate. Got: HashAggregator.
Name: SpilledRecordbatch.java. Expected: SpilledRecordbatch. Got:
SimpleRecordBatch.
Name: StreamingAggTemplate.java. Expected: StreamingAggTemplate. Got:
StreamingAggregator.
Name: SortMemoryManager.java. Expected: SortMemoryManager. Got:
ExternalSortBatch.
Name: SortConfig.java. Expected: SortConfig. Got: ExternalSortBatch.
Name: SortImpl.java. Expected: SortImpl. Got: ExternalSortBatch.
Name: SingleSenderCreator.java. Expected: SingleSenderCreator. Got:
SingleSenderRootExec.
Name: HashTableTemplate.java. Expected: HashTableTemplate. Got: HashTable.
Name: FrameSupportTemplate.java. Expected: FrameSupportTemplate. Got:
NoFrameSupportTemplate.
Name: ScreenCreator.java. Expected: ScreenCreator. Got: ScreenRoot.
Name: UnionAll.java. Expected: UnionAll. Got: Filter.
Name: AvgIntervalTypeFunctions.java. Expected: AvgIntervalTypeFunctions. Got:
AvgFunctions.
Name: PersistedOptionValue.java. Expected: PersistedOptionValue. Got:
Deserializer.
Name: ThreadsResources.java. Expected: ThreadsResources. Got: MetricsResources.
Name: RepeatedVarCharOutput.java. Expected: RepeatedVarCharOutput. Got:
BaseFieldOutput.
Name: MockSubScanPOP.java. Expected: MockSubScanPOP. Got: MockGroupScanPOP.
Name: InMemoryStore.java. Expected: InMemoryStore. Got: InMemoryPersistentStore.
Name: ParquetColumnChunkPageWriteStore.java. Expected:
ParquetColumnChunkPageWriteStore. Got: ParquetDirectByteBufferAllocator.
Name: CorrelationTypeFunctions.java. Expected: CorrelationTypeFunctions. Got:
${aggrtype.className}Functions.
Name: MathFunctionTemplates.java. Expected: MathFunctionTemplates. Got:
${inputType.className}Functions.
Name: CastHigh.java. Expected: CastHigh. Got: CastHighFunctions.
Name: IntervalAggrFunctions2.java. Expected: IntervalAggrFunctions2. Got:
${aggrtype.className}Functions.
Name: SumZeroAggr.java. Expected: SumZeroAggr. Got: SumZeroFunctions.
Name: NumericFunctionsTemplates.java.
[jira] [Resolved] (DRILL-2873) CTAS reports error when timestamp values in CSV file are quoted
[
https://issues.apache.org/jira/browse/DRILL-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-2873.
-
Resolution: Fixed
> CTAS reports error when timestamp values in CSV file are quoted
> ---
>
> Key: DRILL-2873
> URL: https://issues.apache.org/jira/browse/DRILL-2873
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 0.9.0
> Environment: 64e3ec52b93e9331aa5179e040eca19afece8317 | DRILL-2611:
> value vectors should report valid value count | 16.04.2015 @ 13:53:34 EDT
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: 1.17.0
>
>
> When timestamp values are quoted in quotes (") inside a CSV data file, CTAS
> statement reports error.
> Failing CTAS
> {code}
> 0: jdbc:drill:> create table prqFrmCSV02 as select cast(columns[0] as int)
> col_int, cast(columns[1] as bigint) col_bgint, cast(columns[2] as char(10))
> col_char, cast(columns[3] as varchar(18)) col_vchar, cast(columns[4] as
> timestamp) col_tmstmp, cast(columns[5] as date) col_date, cast(columns[6] as
> boolean) col_boln, cast(columns[7] as double) col_dbl from `csvToPrq.csv`;
> Query failed: SYSTEM ERROR: Invalid format: ""2015-04-23 23:47:00.124""
> [a601a66a-b305-4a92-9836-f39edcdc8fe8 on centos-02.qa.lab:31010]
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-04-24 18:41:09,721 [2ac571ba-778f-f3d5-c60f-af2e536905a3:frag:0:0] ERROR
> o.a.drill.exec.ops.FragmentContext - Fragment Context received failure --
> Fragment: 0:0
> org.apache.drill.common.exceptions.DrillUserException: SYSTEM ERROR: Invalid
> format: ""2015-04-23 23:47:00.124""
> [a601a66a-b305-4a92-9836-f39edcdc8fe8 on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.DrillUserException$Builder.build(DrillUserException.java:115)
> ~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.common.exceptions.ErrorHelper.wrap(ErrorHelper.java:39)
> ~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:151)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:131)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:74)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:76)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:64)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:164)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
>
[jira] [Created] (DRILL-7528) Update Avro format plugin documentation
Arina Ielchiieva created DRILL-7528: --- Summary: Update Avro format plugin documentation Key: DRILL-7528 URL: https://issues.apache.org/jira/browse/DRILL-7528 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Currently documentation states that Avro plugin is experimental. As per Drill 1.17 / 1.18 it's code is pretty stable (since Drill 1.18 it uses EVF). Documentation should be updated accordingly. https://drill.apache.org/docs/querying-avro-files/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-5024) CTAS with LIMIT 0 query in SELECT stmt does not create parquet file
[
https://issues.apache.org/jira/browse/DRILL-5024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5024.
-
Resolution: Fixed
> CTAS with LIMIT 0 query in SELECT stmt does not create parquet file
> ---
>
> Key: DRILL-5024
> URL: https://issues.apache.org/jira/browse/DRILL-5024
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.8.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: 1.17.0
>
>
> Note that CTAS was successful
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> create table regtbl_w0rows as select * from
> typeall_l LIMIT 0;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 0 |
> +---++
> 1 row selected (0.51 seconds)
> {noformat}
> But a SELECT on CTAS created file fails.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from regtbl_w0rows;
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 27: Table
> 'regtbl_w0rows' not found
> SQL Query null
> [Error Id: 0569cf98-3800-43ee-b635-aa101b016d46 on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> DROP on the CTAS created table also fails
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> drop table regtbl_w0rows;
> Error: VALIDATION ERROR: Table [regtbl_w0rows] not found
> [Error Id: fb0b1ea8-f76d-42e2-b69c-4beae2798bdf on centos-01.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Verified that CTAS did not create a physical file in dfs.tmp schema
> {noformat}
> [test@cent01 bin]# hadoop fs -ls /tmp/regtbl_w0rows
> ls: `/tmp/regtbl_w0rows': No such file or directory
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-3916) Assembly for JDBC storage plugin missing
[ https://issues.apache.org/jira/browse/DRILL-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-3916. - Resolution: Fixed > Assembly for JDBC storage plugin missing > > > Key: DRILL-3916 > URL: https://issues.apache.org/jira/browse/DRILL-3916 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JDBC, Storage - Other >Affects Versions: 1.2.0 >Reporter: Andrew >Assignee: Andrew >Priority: Major > > The JDBC storage plugin is missing from the assembly instructions, which > means that the plugin fails to be loaded by the drill bit on start. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7526) Assertion Error when only type with used with schema in table function
Arina Ielchiieva created DRILL-7526:
---
Summary: Assertion Error when only type with used with schema in
table function
Key: DRILL-7526
URL: https://issues.apache.org/jira/browse/DRILL-7526
Project: Apache Drill
Issue Type: Bug
Reporter: Arina Ielchiieva
{{org.apache.drill.TestSchemaWithTableFunction}}
{noformat}
@Test
public void testWithTypeAndSchema() {
String query = "select Year from
table(dfs.`store/text/data/cars.csvh`(type=> 'text', " +
"schema=>'inline=(`Year` int)')) where Make = 'Ford'";
queryBuilder().sql(query).print();
}
{noformat}
{noformat}
Caused by: java.lang.AssertionError: BOOLEAN
at
org.apache.calcite.sql.type.SqlTypeExplicitPrecedenceList.compareTypePrecedence(SqlTypeExplicitPrecedenceList.java:140)
at org.apache.calcite.sql.SqlUtil.bestMatch(SqlUtil.java:687)
at
org.apache.calcite.sql.SqlUtil.filterRoutinesByTypePrecedence(SqlUtil.java:656)
at
org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:515)
at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:435)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:240)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:218)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5640)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5627)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1692)
at
org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl(ProcedureNamespace.java:53)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3129)
at
org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3111)
at
org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3383)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:944)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:651)
at
org.apache.drill.exec.planner.sql.conversion.SqlConverter.validate(SqlConverter.java:189)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:648)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:170)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:590)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:275)
... 1 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7525) Convert SequqenceFiles to EVF
Arina Ielchiieva created DRILL-7525: --- Summary: Convert SequqenceFiles to EVF Key: DRILL-7525 URL: https://issues.apache.org/jira/browse/DRILL-7525 Project: Apache Drill Issue Type: Improvement Reporter: Arina Ielchiieva Convert SequqenceFiles to EVF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7504) Upgrade Parquet library to 1.11.0
Arina Ielchiieva created DRILL-7504: --- Summary: Upgrade Parquet library to 1.11.0 Key: DRILL-7504 URL: https://issues.apache.org/jira/browse/DRILL-7504 Project: Apache Drill Issue Type: Task Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Fix For: 1.18.0 Upgrade Parquet library to 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7497) Fix warnings when starting Drill on Windows using Java 11
Arina Ielchiieva created DRILL-7497:
---
Summary: Fix warnings when starting Drill on Windows using Java 11
Key: DRILL-7497
URL: https://issues.apache.org/jira/browse/DRILL-7497
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Vova Vysotskyi
Fix For: 1.18.0
Warnings are displayed in SqlLine when starting Drill in embedded mode on
Windows using Java 11:
{noformat}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions
(file:/C:/drill_1_17/apache-drill-1.17.0/apache-drill-1.17.0/jars/3rdparty/javassist-3.26.0-GA.jar)
to method
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of
javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7492) Do not display Failure node entry if it is not set
Arina Ielchiieva created DRILL-7492:
---
Summary: Do not display Failure node entry if it is not set
Key: DRILL-7492
URL: https://issues.apache.org/jira/browse/DRILL-7492
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/rest/profile/profile.ftl#L223
{{errorNode}} is not set in most the times, we should not display {{Failure
node}} entry is its value is null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7489) NPE for star query with alias
Arina Ielchiieva created DRILL-7489:
---
Summary: NPE for star query with alias
Key: DRILL-7489
URL: https://issues.apache.org/jira/browse/DRILL-7489
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Igor Guzenko
Query where alias is used for star should throw exception.
Query on table with defined schema returns correct exception:
{noformat}
select * col_alias from sys.version;
org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: At
line 1, column 8: Unknown identifier '*'
{noformat}
Query on table with dynamic schema returns NPE:
{noformat}
select * col_alias from cp.`tpch/nation.parquet`
Caused by: java.lang.NullPointerException:
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3690)
at
org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4765)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625)
at
org.apache.calcite.sql2rel.StandardConvertletTable.lambda$new$9(StandardConvertletTable.java:204)
at
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:63)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4756)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4625)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3908)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3150)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563)
at
org.apache.drill.exec.planner.sql.SqlConverter.toRel(SqlConverter.java:381)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:685)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:202)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:282)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:162)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:139)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:92)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:590)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:275)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7488) Complex queries on INFORMATION_SCHEMA fail when `planner.slice_target`is small
Arina Ielchiieva created DRILL-7488:
---
Summary: Complex queries on INFORMATION_SCHEMA fail when
`planner.slice_target`is small
Key: DRILL-7488
URL: https://issues.apache.org/jira/browse/DRILL-7488
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Set small value for slice_target:
{code:sql}
alter session set `planner.slice_target`=1;
{code}
Run query on INFORMATION_SCHEMA:
{code:sql}
select * from information_schema.`tables` where TABLE_NAME='lineitem' order by
TABLE_NAME;
{code}
It will fail with the following exception:
{noformat}
java.lang.Exception: org.apache.drill.common.exceptions.UserRemoteException:
SYSTEM ERROR: UnsupportedOperationException: Schema tree can only be created in
root fragment. This is a non-root fragment.
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7482) Fix "could not find artifact" warning Drill build
Arina Ielchiieva created DRILL-7482: --- Summary: Fix "could not find artifact" warning Drill build Key: DRILL-7482 URL: https://issues.apache.org/jira/browse/DRILL-7482 Project: Apache Drill Issue Type: Bug Affects Versions: 1.16.0 Reporter: Arina Ielchiieva Assignee: Vova Vysotskyi Fix For: 1.17.0 Fix the following warning during Drill build: Could not find artifact org.glassfish:javax.el:pom:3.0.1-b07-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7481) Fix raw type warnings in Iceberg Metastore
Arina Ielchiieva created DRILL-7481: --- Summary: Fix raw type warnings in Iceberg Metastore Key: DRILL-7481 URL: https://issues.apache.org/jira/browse/DRILL-7481 Project: Apache Drill Issue Type: Bug Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Fix raw type warnings in Iceberg Metastore module and related classes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7476) Info in some sys schema tables are missing if queried with limit clause
Arina Ielchiieva created DRILL-7476:
---
Summary: Info in some sys schema tables are missing if queried
with limit clause
Key: DRILL-7476
URL: https://issues.apache.org/jira/browse/DRILL-7476
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Arina Ielchiieva
Affected schema: sys
Affected tables: connections, threads, memory
If query is executed with limit clause, information for some fields are missing:
*Connections*
{noformat}
apache drill (sys)> select * from connections;
+---+---++-+---+-+-+-+--+--+
| user|client | drillbit | established |
duration | queries | isAuthenticated | isEncrypted | usingSSL |
session|
+---+---++-+---+-+-+-+--+--+
| anonymous | xxx.xxx.x.xxx | xxx | 2019-12-10 13:45:01.766 | 59 min 42.393 sec
| 27 | false | false | false| xxx |
+---+---++-+---+-+-+-+--+--+
1 row selected (0.1 seconds)
apache drill (sys)> select * from connections limit 1;
+--++--+-+--+-+-+-+--+-+
| user | client | drillbit | established | duration | queries |
isAuthenticated | isEncrypted | usingSSL | session |
+--++--+-+--+-+-+-+--+-+
| || | 2019-12-10 13:45:01.766 | | 28 |
false | false | false| |
+--++--+-+--+-+-+-+--+-+
{noformat}
*Threads*
{noformat}
apache drill (sys)> select * from threads;
++---+---+--+
| hostname | user_port | total_threads | busy_threads |
++---+---+--+
| xxx | 31010 | 27| 23 |
++---+---+--+
1 row selected (0.119 seconds)
apache drill (sys)> select * from threads limit 1;
+--+---+---+--+
| hostname | user_port | total_threads | busy_threads |
+--+---+---+--+
| | 31010 | 27| 24 |
{noformat}
*Memory*
{noformat}
apache drill (sys)> select * from memory;
++---+--+++++
| hostname | user_port | heap_current | heap_max | direct_current |
jvm_direct_current | direct_max |
++---+--+++++
| xxx | 31010 | 493974480| 4116185088 | 5048576| 122765
| 8589934592 |
++---+--+++++
1 row selected (0.115 seconds)
apache drill (sys)> select * from memory limit 1;
+--+---+--+++++
| hostname | user_port | heap_current | heap_max | direct_current |
jvm_direct_current | direct_max |
+--+---+--+++++
| | 31010 | 499343272| 4116185088 | 9048576| 122765
| 8589934592 |
+--+---+--+++++
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7454) Convert the Avro format plugin plugin to use EVF
Arina Ielchiieva created DRILL-7454: --- Summary: Convert the Avro format plugin plugin to use EVF Key: DRILL-7454 URL: https://issues.apache.org/jira/browse/DRILL-7454 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.17.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.18.0 Convert the Avro format plugin plugin to use EVF. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-6723) Kafka reader fails on malformed JSON
[
https://issues.apache.org/jira/browse/DRILL-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-6723.
-
Resolution: Fixed
Merged with commit id ffd3c352586a5884747edcc5a93b0c625a47e100 in the scope of
DRILL-7388.
> Kafka reader fails on malformed JSON
>
>
> Key: DRILL-6723
> URL: https://issues.apache.org/jira/browse/DRILL-6723
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
>Affects Versions: 1.14.0
> Environment: java version "1.8.0_91"
> Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
> $ bin/drill-embedded
> Aug 30, 2018 5:29:08 PM org.glassfish.jersey.server.ApplicationHandler
> initialize
> INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29
> 01:25:26...
> apache drill 1.14.0
>
>Reporter: Matt Keranen
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.17.0
>
>
> With a Kakfa topic where the first message is a simple string "test", and
> store.json.reader.skip_invalid_records = true, a SELECT * FROM topic still
> fails following error.
> In addition, using an OFFSET does not appear to allow the bad messages to be
> bypassed. Same error occurs on the first message.
> {noformat}
> 0: jdbc:drill:zk=local> select * from kafka.`logs` limit 5;
> Error: DATA_READ ERROR: Failure while reading messages from kafka.
> Recordreader was at record: 1
> Not a JSON Object: "TEST"
> Fragment 0:0
> [Error Id: 965d7a69-3d77-4a11-9613-3892a95c4a63 on x.x.x.x:31010]
> (state=,code=0)
> {noformat}
> Description:
> New option {{store.kafka.reader.skip_invalid_records}} will be introduced to
> cover this case. Default is false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-6739) Update Kafka libs to 2.0.0+ version
[
https://issues.apache.org/jira/browse/DRILL-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-6739.
-
Resolution: Fixed
Merged with commit id ffd3c352586a5884747edcc5a93b0c625a47e100 in the scope of
DRILL-7388.
> Update Kafka libs to 2.0.0+ version
> ---
>
> Key: DRILL-6739
> URL: https://issues.apache.org/jira/browse/DRILL-6739
> Project: Apache Drill
> Issue Type: Task
> Components: Storage - Kafka
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.17.0
>
>
> The current version of Kafka libs is 0.11.0.1
> The last version is 2.0.0 (September 2018)
> https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients
> Looks like the only changes which should be done are:
> * replacing {{serverConfig()}} method with {{staticServerConfig()}} in Drill
> {{EmbeddedKafkaCluster}} class
> * Replacing deprecated {{AdminUtils}} with {{kafka.zk.AdminZkClient}}
> [https://github.com/apache/kafka/blob/3cdc78e6bb1f83973a14ce1550fe3874f7348b05/core/src/main/scala/kafka/admin/AdminUtils.scala#L35]
> https://issues.apache.org/jira/browse/KAFKA-6545
> The initial work: https://github.com/vdiravka/drill/commits/DRILL-6739
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-7290) “Failed to construct kafka consumer” using Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-7290.
-
Resolution: Fixed
Merged with commit id ffd3c352586a5884747edcc5a93b0c625a47e100 in the scope of
DRILL-7388.
> “Failed to construct kafka consumer” using Apache Drill
> ---
>
> Key: DRILL-7290
> URL: https://issues.apache.org/jira/browse/DRILL-7290
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0, 1.16.0
>Reporter: Aravind Voruganti
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.17.0
>
>
> {noformat}
> {noformat}
> I am using the Apache Drill (1.14) JDBC driver in my application which
> consumes the data from the Kafka. The application works just fine for some
> time and after few iterations it fails to execute due to the following *Too
> many files open* issue. I made sure there are no file handle leaks in my code
> but still nor sure why this issue is happening?
>
> It looks like the issue is happening from with-in the Apache drill libraries
> when constructing the Kafka consumer. Can any one please guide me help this
> problem fixed?
> The problem perishes when I restart my Apache drillbit but very soon it
> happens again. I did check the file descriptor count on my unix machine using
> *{{ulimit -a | wc -l}} & {{lsof -a -p | wc -l}}* before and after the
> drill process restart and it seems the drill process is considerably taking a
> lot of file descriptors. I tried increasing the file descriptor count on the
> system but still no luck.
> I have followed the Apache Drill storage plugin documentation in configuring
> the Kafka plugin into Apache Drill at
> [https://drill.apache.org/docs/kafka-storage-plugin/]
> Any help on this issue is highly appreciated. Thanks.
> JDBC URL: *{{jdbc:drill:drillbit=localhost:31010;schema=kafka}}*
> NOTE: I am pushing down the filters in my query {{SELECT * FROM myKafkaTopic
> WHERE kafkaMsgTimestamp > 1560210931626}}
>
> 2019-06-11 08:43:13,639 [230033ed-d410-ae7c-90cb-ac01d3b404cc:foreman] INFO
> o.a.d.e.store.kafka.KafkaGroupScan - User Error Occurred: Failed to fetch
> start/end offsets of the topic myKafkaTopic (Failed to construct kafka
> consumer)
> org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Failed to
> fetch start/end offsets of the topic myKafkaTopic
> Failed to construct kafka consumer
> [Error Id: 73f896a7-09d4-425b-8cd5-f269c3a6e69a ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.store.kafka.KafkaGroupScan.init(KafkaGroupScan.java:198)
> [drill-storage-kafka-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.store.kafka.KafkaGroupScan.(KafkaGroupScan.java:98)
> [drill-storage-kafka-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.store.kafka.KafkaStoragePlugin.getPhysicalScan(KafkaStoragePlugin.java:83)
> [drill-storage-kafka-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:111)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:89)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:69)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:62)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:38)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:652)
> [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
> [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:429)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:369)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:255)
> [drill-java-exec-1.14.0.jar:1.14.0]
> at
>
[jira] [Created] (DRILL-7448) Fix warnings when running Drill memory tests
Arina Ielchiieva created DRILL-7448:
---
Summary: Fix warnings when running Drill memory tests
Key: DRILL-7448
URL: https://issues.apache.org/jira/browse/DRILL-7448
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Bohdan Kazydub
Fix For: 1.17.0
{noformat}
-- drill-memory-base
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.drill.exec.memory.TestEndianess
[INFO] Running org.apache.drill.exec.memory.TestAccountant
16:21:45,719 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
NOT find resource [logback.groovy]
16:21:45,719 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [logback-test.xml] at
[jar:file:/Users/arina/Development/git_repo/drill/common/target/drill-common-1.17.0-SNAPSHOT-tests.jar!/logback-test.xml]
16:21:45,733 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@dbd940d - URL
[jar:file:/Users/arina/Development/git_repo/drill/common/target/drill-common-1.17.0-SNAPSHOT-tests.jar!/logback-test.xml]
is not of type file
16:21:45,780 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- debug attribute not set
16:21:45,802 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
16:21:45,802 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
16:21:45,803 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
16:21:45,811 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [STDOUT]
16:21:45,826 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
16:21:45,866 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT
level set to ERROR
16:21:45,866 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
16:21:45,866 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
16:21:45,866 |-WARN in ch.qos.logback.classic.joran.action.RootLoggerAction -
The object on the top the of the stack is not the root logger
16:21:45,866 |-WARN in ch.qos.logback.classic.joran.action.RootLoggerAction -
It is: ch.qos.logback.core.joran.conditional.IfAction
16:21:45,866 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- End of configuration.
16:21:45,867 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@71d15f18
- Registering current configuration as safe fallback point
16:21:45,717 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
NOT find resource [logback.groovy]
16:21:45,717 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [logback-test.xml] at
[jar:file:/Users/arina/Development/git_repo/drill/common/target/drill-common-1.17.0-SNAPSHOT-tests.jar!/logback-test.xml]
16:21:45,729 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@2698dc7 - URL
[jar:file:/Users/arina/Development/git_repo/drill/common/target/drill-common-1.17.0-SNAPSHOT-tests.jar!/logback-test.xml]
is not of type file
16:21:45,778 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- debug attribute not set
16:21:45,807 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
16:21:45,807 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
16:21:45,808 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
16:21:45,814 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [STDOUT]
16:21:45,829 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
16:21:45,868 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT
level set to ERROR
16:21:45,868 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
16:21:45,868 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
16:21:45,868 |-WARN in ch.qos.logback.classic.joran.action.RootLoggerAction -
The object on the top the of the stack is not the root logger
16:21:45,868 |-WARN in
[jira] [Resolved] (DRILL-5506) Apache Drill Querying data from compressed .zip file
[ https://issues.apache.org/jira/browse/DRILL-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5506. - Resolution: Fixed Fixed in the scope of DRILL-5674. > Apache Drill Querying data from compressed .zip file > > > Key: DRILL-5506 > URL: https://issues.apache.org/jira/browse/DRILL-5506 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Affects Versions: 1.10.0 >Reporter: john li >Priority: Major > Labels: newbie > Fix For: 1.17.0 > > > Referring to the previous issue > https://issues.apache.org/jira/browse/DRILL-2806 > According to the remarks from Steven Phillips added a comment - 16/Apr/15 > 21:50 > "The only compression codecs that work with Drill out of the box are gz, and > bz2. Additional codecs can be added by including the relevant libraries in > the Drill classpath." > I would like to learn how to use Apache Drill to query data from compressed > .zip file. > > However , the only default compression codecs that work with Apache Drill are > gz, and bz2. > > Assuming that Additional codecs can be added by including the relevant > libraries in the Drill classpath. > > Please kindly show me the step by step instructions so that I can understand > how exactly to add the "zip" codec and how to include the relevant libraries > in the Drill classpath ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7419) Enhance Drill splitting logic for compressed files
Arina Ielchiieva created DRILL-7419: --- Summary: Enhance Drill splitting logic for compressed files Key: DRILL-7419 URL: https://issues.apache.org/jira/browse/DRILL-7419 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.16.0 Reporter: Arina Ielchiieva By default Drill treats all compressed files are non splittable. Drill uses BlockMapBuilder to split file into blocks if possible. According to its code, it tries to split the file if blockSplittable is set to true and file IS NOT compressed. So even if format is block splittable but came as compressed file, it won't be split. https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/BlockMapBuilder.java#L115 But some compression codecs can be splittable, for example; bzip2 (https://i.stack.imgur.com/jpprr.jpg). Codec type should be taken into account when considering if file can be split. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-5436) Need a way to input password which contains space when calling sqlline
[
https://issues.apache.org/jira/browse/DRILL-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5436.
-
Resolution: Fixed
Fixed in the scope of https://issues.apache.org/jira/browse/DRILL-7401.
> Need a way to input password which contains space when calling sqlline
> --
>
> Key: DRILL-5436
> URL: https://issues.apache.org/jira/browse/DRILL-5436
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - CLI
>Affects Versions: 1.10.0
>Reporter: Hao Zhu
>Priority: Major
> Fix For: 1.17.0
>
>
> create a user named "spaceuser" with password "hello world".
> All below failed:
> {code}
> sqlline -u jdbc:drill:zk=xxx -n spaceuser -p 'hello world'
> sqlline -u jdbc:drill:zk=xxx -n spaceuser -p "hello world"
> sqlline -u jdbc:drill:zk=xxx -n spaceuser -p 'hello\ world'
> sqlline -u jdbc:drill:zk=xxx -n spaceuser -p "hello\ world"
> {code}
> Need a way to input password which contains space when calling sqlline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7418) MetadataDirectGroupScan improvements
Arina Ielchiieva created DRILL-7418:
---
Summary: MetadataDirectGroupScan improvements
Key: DRILL-7418
URL: https://issues.apache.org/jira/browse/DRILL-7418
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.17.0
When count is converted to direct scan (case when statistics and table metadata
are available and there is no need to perform count operation),
{{MetadataDirectGroupScan}} is used. Proposed {{MetadataDirectGroupScan}}
enhancements:
1. show table root instead listing all table files. If users= has lots of
files, query plan gets polluted with files enumeration. Since files are not
used for calculation (only metadata), they are not relevant and can be excluded
from plan.
Before:
{noformat}
| 00-00Screen
00-01 Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[$2], EXPR$3=[$3])
00-02DirectScan(groupscan=[files =
[/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_0.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_5.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_4.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_9.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_3.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_6.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_7.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_10.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_2.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_1.parquet,
/drill/testdata/metadata_cache/store_sales_null_blocks_all/0_0_8.parquet],
numFiles = 11, usedMetadataSummaryFile = false, DynamicPojoRecordReader{records
= [[1560060, 2880404, 2880404, 0]]}])
{noformat}
After:
{noformat}
| 00-00Screen
00-01 Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[$2], EXPR$3=[$3])
00-02DirectScan(groupscan=[selectionRoot =
/drill/testdata/metadata_cache/store_sales_null_blocks_all, numFiles = 11,
usedMetadataSummaryFile = false, DynamicPojoRecordReader{records = [[1560060,
2880404, 2880404, 0]]}])
{noformat}
2. Submission of physical plan which contains {{MetadataDirectGroupScan}} fails
with deserialization errors, proper ser / de should be implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-6842) Export to CSV using CREATE TABLE AS (CTAS) wrong parsed
[
https://issues.apache.org/jira/browse/DRILL-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-6842.
-
Resolution: Fixed
Fixed in the scope of DRILL-6096.
> Export to CSV using CREATE TABLE AS (CTAS) wrong parsed
> ---
>
> Key: DRILL-6842
> URL: https://issues.apache.org/jira/browse/DRILL-6842
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Text CSV, Storage - Writer
>Affects Versions: 1.14.0
> Environment: - Tested with latest version *Apache Drill* 1.14.0, and
> building the latest version from master (Github repo), commit
> ad61c6bc1dd24994e50fe7dfed043d5e57dba8f9 at _Nov 5, 2018_.
> - *Linux* x64, Ubuntu 16.04
> - *OpenJDK* Runtime Environment (build
> 1.8.0_171-8u171-b11-0ubuntu0.17.10.1-b11)
> - Apache *Maven* 3.5.0
>Reporter: Mariano Ruiz
>Priority: Minor
> Labels: csv, export
> Fix For: 1.17.0
>
> Attachments: Screenshot from 2018-11-09 14-18-43.png
>
>
> When you export to a CSV using CTAS the result of a query, most of the time
> the generated file is OK, but if you have in the results text columns with
> "," characters, the resulting CSV file is broken, because does not enclose
> the cells with commas inside with the " character.
> Steps to reproduce the bug:
> Lets say you have the following table in some source of data, maybe a CSV
> file too:
> {code:title=/tmp/input.csv}
> product_ean,product_name,product_brand
> 12345678900,IPhone X,Apple
> 9911100,"Samsung S9, Black",Samsung
> 1223456,Smartwatch XY,Some Brand
> {code}
> Note that the second row of data, in the column "product_name", it has a
> value with a comma inside (_Samsung S9, Black_), so all the cell value is
> enclosed with " characters, while the rest of the column cells aren't,
> despite they could be enclosed too.
> So if you query this file, Drill will interpret correctly the file and does
> not interpret that comma inside the cell as a separator like the rest of the
> commas in the file:
> {code}
> 0: jdbc:drill:zk=local> SELECT * FROM dfs.`/tmp/input.csv`;
> +--+++
> | product_ean |product_name| product_brand |
> +--+++
> | 12345678900 | IPhone X | Apple |
> | 9911100 | Samsung S9, Black | Samsung|
> | 1223456 | Smartwatch XY | Some Brand |
> +--+++
> 3 rows selected (1.874 seconds)
> {code}
> But now, if you want to query the file and export the result as CSV using the
> CTAS feature, using the following steps:
> {code}
> 0: jdbc:drill:zk=local> USE dfs.tmp;
> +---+--+
> | ok | summary|
> +---+--+
> | true | Default schema changed to [dfs.tmp] |
> +---+--+
> 1 row selected (0.13 seconds)
> 0: jdbc:drill:zk=local> ALTER SESSION SET `store.format`='csv';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.094 seconds)
> 0: jdbc:drill:zk=local> CREATE TABLE dfs.tmp.my_output AS SELECT * FROM
> dfs.`/tmp/input.csv`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 3 |
> +---++
> 1 row selected (0.453 seconds)
> {code}
> The output file is this:
> {code:title=/tmp/my_output/0_0_0.csv}
> product_ean,product_name,product_brand
> 12345678900,IPhone X,Apple
> 9911100,Samsung S9, Black,Samsung
> 1223456,Smartwatch XY,Some Brand
> {code}
> The text _Samsung S9, Black_ in the cell is not quoted, so any CSV
> interpreter like an office tool, a Java/Python/... library will interpret it
> as two cell instead of one. Even Apache Drill will interpret it wrong:
> {code}
> 0: jdbc:drill:zk=local> SELECT * FROM dfs.`/tmp/my_output/0_0_0.csv`;
> +--+++
> | product_ean | product_name | product_brand |
> +--+++
> | 12345678900 | IPhone X | Apple |
> | 9911100 | Samsung S9 | Black |
> | 1223456 | Smartwatch XY | Some Brand |
> +--+++
> 3 rows selected (0.175 seconds)
> {code}
> Note that the ending part _ Black_ was interpreted as a following cell, and
> the real following cell is not showed, but it's not an error in the Drill
> interpreter, it's an error of how
[jira] [Resolved] (DRILL-4788) Exporting from Parquet to CSV - commas in strings are not escaped
[ https://issues.apache.org/jira/browse/DRILL-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-4788. - Resolution: Fixed Fixed in the scope of DRILL-6096. > Exporting from Parquet to CSV - commas in strings are not escaped > - > > Key: DRILL-4788 > URL: https://issues.apache.org/jira/browse/DRILL-4788 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.6.0 > Environment: Linux >Reporter: Richard Patching >Priority: Major > Labels: csv, csvparser, export > Fix For: 1.17.0 > > > When exporting data from Parquet to CSV, if there is a column which contains > a comma, the text after the comma gets put into the next column instead of > being escaped. > The only work around is to do REGEXP_REPLACE(COLUMN[0], ',',' ') which > replaced the comma in the string with a blank space. This is not ideal in > terms of keeping a true accurate record of the data we receive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-6958) CTAS csv with option
[ https://issues.apache.org/jira/browse/DRILL-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-6958. - Resolution: Fixed Fixed in the scope of DRILL-6096. > CTAS csv with option > > > Key: DRILL-6958 > URL: https://issues.apache.org/jira/browse/DRILL-6958 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Text CSV >Affects Versions: 1.15.0, 1.16.0 >Reporter: benj >Priority: Major > Fix For: 1.17.0 > > > Currently, it may be difficult to produce well-formed CSV with CTAS (see > comment below). > It appears necessary to have some additional/configuratble options to write > CSV file with CTAS : > * possibility to change/define the separator, > * possibility to write or not the header, > * possibility to force the write of only 1 file instead of lot of parts, > * possibility to force quoting > * possibility to use/change escape char > * ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-3850) Execute multiple commands from sqlline -q
[
https://issues.apache.org/jira/browse/DRILL-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-3850.
-
Resolution: Fixed
Fixed in the scope of DRILL-7401.
> Execute multiple commands from sqlline -q
> -
>
> Key: DRILL-3850
> URL: https://issues.apache.org/jira/browse/DRILL-3850
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - CLI
>Affects Versions: 1.1.0, 1.2.0
> Environment: Mint 17.1
>Reporter: Philip Deegan
>Priority: Major
> Fix For: 1.17.0
>
>
> Be able to perform
> {noformat}
> ./sqlline -u jdbc:drill:zk=local -q "use dfs.tmp; alter session set
> \`store.format\`='csv';"
> {noformat}
> instead of
> {noformat}
> ./sqlline -u jdbc:drill:zk=local -q "use dfs.tmp;"
> ./sqlline -u jdbc:drill:zk=local -q "alter session set
> \`store.format\`='csv';"
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-7195) Query returns incorrect result or does not fail when cast with is null is used in filter condition
[
https://issues.apache.org/jira/browse/DRILL-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-7195.
-
Resolution: Fixed
> Query returns incorrect result or does not fail when cast with is null is
> used in filter condition
> --
>
> Key: DRILL-7195
> URL: https://issues.apache.org/jira/browse/DRILL-7195
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> 1. For the case when a query contains filter with a {{cast}} which cannot be
> done with {{is null}}, the query does not fail:
> {code:sql}
> select * from dfs.tmp.`a.json` as t where cast(t.a as integer) is null;
> +---+
> | a |
> +---+
> +---+
> No rows selected (0.142 seconds)
> {code}
> where
> {noformat}
> cat /tmp/a.json
> {"a":"aaa"}
> {noformat}
> But for the case when this condition is specified in project, query, as it is
> expected, fails:
> {code:sql}
> select cast(t.a as integer) is null from dfs.tmp.`a.json` t;
> Error: SYSTEM ERROR: NumberFormatException: aaa
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: ed3982ce-a12f-4d63-bc6e-cafddf28cc24 on user515050-pc:31010]
> (state=,code=0)
> {code}
> This is a regression, for Drill 1.15 the first and the second queries are
> failed:
> {code:sql}
> select * from dfs.tmp.`a.json` as t where cast(t.a as integer) is null;
> Error: SYSTEM ERROR: NumberFormatException: aaa
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: 2f878f15-ddaa-48cd-9dfb-45c04db39048 on user515050-pc:31010]
> (state=,code=0)
> {code}
> 2. For the case when {{drill.exec.functions.cast_empty_string_to_null}} is
> enabled, this issue will cause wrong results:
> {code:sql}
> alter system set `drill.exec.functions.cast_empty_string_to_null`=true;
> select * from dfs.tmp.`a1.json` t where cast(t.a as integer) is null;
> +---+
> | a |
> +---+
> +---+
> No rows selected (1.759 seconds)
> {code}
> where
> {noformat}
> cat /tmp/a1.json
> {"a":"1"}
> {"a":""}
> {noformat}
> Result for Drill 1.15.0:
> {code:sql}
> select * from dfs.tmp.`a1.json` t where cast(t.a as integer) is null;
> ++
> | a |
> ++
> ||
> ++
> 1 row selected (1.724 seconds)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7401) Sqlline 1.9 upgrade
Arina Ielchiieva created DRILL-7401: --- Summary: Sqlline 1.9 upgrade Key: DRILL-7401 URL: https://issues.apache.org/jira/browse/DRILL-7401 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Upgrade to SqlLine 1.9 once it is released (https://github.com/julianhyde/sqlline/issues/350). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-5929) Misleading error for text file with blank line delimiter
[
https://issues.apache.org/jira/browse/DRILL-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5929.
-
Resolution: Fixed
Fixed with V3 reader introduction. Error message is the following:
{{org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: The
text format line delimiter cannot be blank.}}
> Misleading error for text file with blank line delimiter
>
>
> Key: DRILL-5929
> URL: https://issues.apache.org/jira/browse/DRILL-5929
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.17.0
>
>
> Consider the following functional test query:
> {code}
> select * from
> table(`table_function/colons.txt`(type=>'text',lineDelimiter=>'\\'))
> {code}
> For some reason (yet to be determined), when running this from Java, the line
> delimiter ended up empty. This cases the following line to fail with an
> {{ArrayIndexOutOfBoundsException}}:
> {code}
> class TextInput ...
> public final byte nextChar() throws IOException {
> if (byteChar == lineSeparator[0]) { // but, lineSeparator.length == 0
> {code}
> We then translate the exception:
> {code}
> class TextReader ...
> public final boolean parseNext() throws IOException {
> ...
> } catch (Exception ex) {
> try {
> throw handleException(ex);
> ...
> private TextParsingException handleException(Exception ex) throws
> IOException {
> ...
> if (ex instanceof ArrayIndexOutOfBoundsException) {
> // Not clear this exception is still thrown...
> ex = UserException
> .dataReadError(ex)
> .message(
> "Drill failed to read your text file. Drill supports up to %d
> columns in a text file. Your file appears to have more than that.",
> MAXIMUM_NUMBER_COLUMNS)
> .build(logger);
> }
> {code}
> That is, due to a missing delimiter, we get an index out of bounds exception,
> which we translate to an error about having too many fields. But, the file
> itself has only a handful of fields. Thus, the error is completely wrong.
> Then, we compound the error:
> {code}
> private TextParsingException handleException(Exception ex) throws
> IOException {
> ...
> throw new TextParsingException(context, message, ex);
> class CompliantTextReader ...
> public boolean next() {
> ...
> } catch (IOException | TextParsingException e) {
> throw UserException.dataReadError(e)
> .addContext("Failure while reading file %s. Happened at or shortly
> before byte position %d.",
> split.getPath(), reader.getPos())
> .build(logger);
> {code}
> That is, our AIOB exception became a user exception that became a text
> parsing exception that became a data read error.
> But, this is not a data read error. It is an error in Drill's own validation
> logic. Not clear we should be wrapping user exceptions in other errors that
> we wrap in other user exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-6984) from table escape parameter not deleted when defined with value other than '"'
[
https://issues.apache.org/jira/browse/DRILL-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-6984.
-
Resolution: Fixed
Issue was fixed with v3 text reader introduction and several subsequent fixes.
> from table escape parameter not deleted when defined with value other than
> '"'
> ---
>
> Key: DRILL-6984
> URL: https://issues.apache.org/jira/browse/DRILL-6984
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Minor
> Fix For: 1.17.0
>
>
> When change the escape char, it's kept in the output instead of be deleted
> like with '"'
> Example :
> the file sample.csv :
> {code:java}
> name|age|subject|info
> Bob|21|"lovely quotes"|none
> Bill|23|"character @"|@" pipe"|none
> {code}
> {code:java}
> SELECT * FROM table(tmp`sample.csv`(`escape`=>'@', type => 'text',
> fieldDelimiter => '|',quote=>'"', extractHeader => true));
> {code}
> The result is
> {code:java}
> | name | age | subject | info |
> +-+--+---+---+
> | Bob | 21 | lovely quotes | none |
> | Bill | 23 | character @"|@" pipe | none |
> {code}
> As we expect : < character "|" pipe > (without the escape char (@)
>
> Note that we have the good behavior when using quote ('"') as escaping
> character
> {code:java}
> name|age|subject|info
> Bob|21|"lovely quotes"|none
> Bill|23|"character ""|"" pipe"|none
> {code}
> {code:java}
> SELECT * FROM table(tmp`sample.csv`(`escape`=>'"', type => 'text',
> fieldDelimiter => '|',quote=>'"', extractHeader => true));
> OR
> SELECT * FROM table(tmp`sample.csv`(type => 'text', fieldDelimiter =>
> '|',quote=>'"', extractHeader => true));
> {code}
> The result is OK with
> {code:java}
> | name | age | subject | info |
> +-+--+---+---+
> | Bob | 21 | lovely quotes | none |
> | Bill| 23 | character "|" pipe | none |
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-4195) NullPointerException received on ResultSet.next() call for query
[ https://issues.apache.org/jira/browse/DRILL-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-4195. - Resolution: Cannot Reproduce > NullPointerException received on ResultSet.next() call for query > > > Key: DRILL-4195 > URL: https://issues.apache.org/jira/browse/DRILL-4195 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC >Affects Versions: 1.2.0 > Environment: Linux lnxx64r6 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May > 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux >Reporter: Sergio Lob >Priority: Major > Attachments: cre8table_trdmix91.sql, setup, test.class, test.java, > test.log, trdmix91.csv > > > NullPointerException received on ResultSet.next() call for a particular > query. We have several different queries that produce NullPointerException. > One is: > "SELECT T1.`fa02int` AS `SK001`, MAX(T1.`fa06char_5`) FROM > `hive`.`default`.`trdmix91` `T1` GROUP BY T1.`fa02int` ORDER BY `SK001`" > During invocation of rs.ext(), I receive the following > exception stack trace: > invoking ResultSet.next() to get to first row: > Exception: java.sql.SQLException: SYSTEM ERROR: NullPointerException > Fragment 0:0 > [Error Id: e7dc2d6e-ab32-4d6d-a593-7fe09a677393 on maprdemo:31010] > java.sql.SQLException: SYSTEM ERROR: NullPointerException > Fragment 0:0 > [Error Id: e7dc2d6e-ab32-4d6d-a593-7fe09a677393 on maprdemo:31010] > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor. > java:247) > at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:320) > at > oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.ja > va:187) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl > .java:160) > at test.main(test.java:64) > Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: > SYSTEM E > RROR: NullPointerException > Fragment 0:0 > [Error Id: e7dc2d6e-ab32-4d6d-a593-7fe09a677393 on maprdemo:31010] > at > oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived( > QueryResultHandler.java:118) > at > oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClie > nt.java:110) > at > oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(Basic > ClientWithConnection.java:47) > at > oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(Basic > ClientWithConnection.java:32) > at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) > at > oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.ja > va:233) > at > oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.ja > va:205) > at > oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(Messa > geToMessageDecoder.java:89) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead > (AbstractChannelHandlerContext.java:339) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(A > bstractChannelHandlerContext.java:324) > at > oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateH > andler.java:254) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead > (AbstractChannelHandlerContext.java:339) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(A > bstractChannelHandlerContext.java:324) > at > oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(Messa > geToMessageDecoder.java:103) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead > (AbstractChannelHandlerContext.java:339) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(A > bstractChannelHandlerContext.java:324) > at > oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMe > ssageDecoder.java:242) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead > (AbstractChannelHandlerContext.java:339) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(A > bstractChannelHandlerContext.java:324) > at > oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(Channe > lInboundHandlerAdapter.java:86) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead > (AbstractChannelHandlerContext.java:339) > at > oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(A > bstractChannelHandlerContext.java:324) > at > oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultC > hannelPipeline.java:847) > at >
[jira] [Resolved] (DRILL-5557) java.lang.IndexOutOfBoundsException: writerIndex:
[ https://issues.apache.org/jira/browse/DRILL-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5557. - Resolution: Fixed > java.lang.IndexOutOfBoundsException: writerIndex: > -- > > Key: DRILL-5557 > URL: https://issues.apache.org/jira/browse/DRILL-5557 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: renlu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (DRILL-5492) CSV reader does not validate header names, causes nonsense output
[
https://issues.apache.org/jira/browse/DRILL-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5492.
-
Resolution: Fixed
> CSV reader does not validate header names, causes nonsense output
> -
>
> Key: DRILL-5492
> URL: https://issues.apache.org/jira/browse/DRILL-5492
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Consider the same test case as in DRILL-5491, but with a slightly different
> input file:
> {code}
> ___
> a,b,c
> d,e,f
> {code}
> The underscores represent three spaces: use spaces in the real test.
> In this case, the code discussed in DRILL-5491 finds some characters and
> happily returns the following array:
> {code}
> [" "]
> {code}
> The field name of three blanks is returned to the client to produce the
> following bizarre output:
> {code}
> 2 row(s):
>
> a
> d
> {code}
> The blank line is normally the header, but the header here was considered to
> be three blanks. (In fact, the blanks are actually printed.)
> Since the blanks were considered to be a field, the file is assumed to have
> only one field, so only the first column was returned.
> The expected behavior is that spaces are trimmed from field names, so the
> field name list would be empty and a User Error thrown. (That is, it is
> confusing to the user why a blank line produces NPE, some produce the
> {{ExecutionSetupException}} shown in DRILL-5491, and some produce blank
> headings. Behavior should be consistent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-5491) NPE when reading a CSV file, with headers, but blank header line
[
https://issues.apache.org/jira/browse/DRILL-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5491.
-
Resolution: Fixed
> NPE when reading a CSV file, with headers, but blank header line
>
>
> Key: DRILL-5491
> URL: https://issues.apache.org/jira/browse/DRILL-5491
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.17.0
>
>
> See DRILL-5490 for background.
> Try this unit test case:
> {code}
> FixtureBuilder builder = ClusterFixture.builder()
> .maxParallelization(1);
> try (ClusterFixture cluster = builder.build();
> ClientFixture client = cluster.clientFixture()) {
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> cluster.defineWorkspace("dfs", "data", "/tmp/data", "csv", csvFormat);
> String sql = "SELECT * FROM `dfs.data`.`csv/test7.csv`";
> client.queryBuilder().sql(sql).printCsv();
> }
> }
> {code}
> The test can also be run as a query using your favorite client.
> Using this input file:
> {code}
> a,b,c
> d,e,f
> {code}
> (The first line is blank.)
> The following is the result:
> {code}
> Exception (no rows returned):
> org.apache.drill.common.exceptions.UserRemoteException:
> SYSTEM ERROR: NullPointerException
> {code}
> The {{RepeatedVarCharOutput}} class tries (but fails for the reasons outlined
> in DRILL-5490) to detect this case.
> The code crashes here in {{CompliantTextRecordReader.extractHeader()}}:
> {code}
> String [] fieldNames = ((RepeatedVarCharOutput)hOutput).getTextOutput();
> {code}
> Because of bad code in {{RepeatedVarCharOutput.getTextOutput()}}:
> {code}
> public String [] getTextOutput () throws ExecutionSetupException {
> if (recordCount == 0 || fieldIndex == -1) {
> return null;
> }
> if (this.recordStart != characterData) {
> throw new ExecutionSetupException("record text was requested before
> finishing record");
> }
> {code}
> Since there is no text on the line, special code elsewhere (see DRILL-5490)
> elects not to increment the {{recordCount}}. (BTW: {{recordCount}} is the
> total across-batch count, probably the in-batch count, {{batchIndex}}, was
> wanted here.) Since the count is zero, we return null.
> But, if the author probably thought we'd get a zero-length record, and the
> if-statement throws an exception in this case. But, see DRILL-5490 about why
> this code does not actually work.
> The result is one bug (not incrementing the record count), triggering another
> (returning a null), which masks a third ({{recordStart}} is not set correctly
> so the exception would not be thrown.)
> All that bad code is just fun and games until we get an NPE, however.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-5451) Query on csv file w/ header fails with an exception when non existing column is requested if file is over 4096 lines long
[
https://issues.apache.org/jira/browse/DRILL-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5451.
-
Resolution: Fixed
> Query on csv file w/ header fails with an exception when non existing column
> is requested if file is over 4096 lines long
> -
>
> Key: DRILL-5451
> URL: https://issues.apache.org/jira/browse/DRILL-5451
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text CSV
>Affects Versions: 1.10.0
> Environment: Tested on CentOs 7 and Ubuntu
>Reporter: Paul Wilson
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: 4097_lines.csvh
>
>
> When querying a text (csv) file with extractHeaders set to true, selecting a
> non existent column works as expected (returns "empty" value) when file has
> 4096 lines or fewer (1 header plus 4095 data), but results in an
> IndexOutOfBoundsException where the file has 4097 lines or more.
> With Storage config:
> {code:javascript}
> "csvh": {
> "type": "text",
> "extensions": [
> "csvh"
> ],
> "extractHeader": true,
> "delimiter": ","
> }
> {code}
> In the following 4096_lines.csvh has is identical to 4097_lines.csvh with the
> last line removed.
> Results:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.`/test/4097_lines.csvh` LIMIT 2;
> +--++
> | line_no |line_description|
> +--++
> | 2| this is line number 2 |
> | 3| this is line number 3 |
> +--++
> 2 rows selected (2.455 seconds)
> 0: jdbc:drill:zk=local> select line_no, non_existent_field from
> dfs.`/test/4096_lines.csvh` LIMIT 2;
> +--+-+
> | line_no | non_existent_field |
> +--+-+
> | 2| |
> | 3| |
> +--+-+
> 2 rows selected (2.248 seconds)
> 0: jdbc:drill:zk=local> select line_no, non_existent_field from
> dfs.`/test/4097_lines.csvh` LIMIT 2;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 16384, length: 4
> (expected: range(0, 16384))
> Fragment 0:0
> [Error Id: eb0974a8-026d-4048-9f10-ffb821a0d300 on localhost:31010]
> (java.lang.IndexOutOfBoundsException) index: 16384, length: 4 (expected:
> range(0, 16384))
> io.netty.buffer.DrillBuf.checkIndexD():123
> io.netty.buffer.DrillBuf.chk():147
> io.netty.buffer.DrillBuf.getInt():520
> org.apache.drill.exec.vector.UInt4Vector$Accessor.get():358
> org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount():659
> org.apache.drill.exec.physical.impl.ScanBatch.next():234
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext():115
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
>
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1657
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> This seems similar to the
[jira] [Created] (DRILL-7397) Fix logback errors when building the project
Arina Ielchiieva created DRILL-7397:
---
Summary: Fix logback errors when building the project
Key: DRILL-7397
URL: https://issues.apache.org/jira/browse/DRILL-7397
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Bohdan Kazydub
{noformat}
[INFO] Compiling 75 source files to /.../drill/common/target/classes
[WARNING] Unable to autodetect 'javac' path, using 'javac' from the environment.
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default) @ drill-common ---
17:46:05,674 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
NOT find resource [logback.groovy]
17:46:05,675 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [logback-test.xml] at
[file:/.../drill/common/src/test/resources/logback-test.xml]
17:46:05,712 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- debug attribute not set
17:46:05,714 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
17:46:05,714 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
17:46:05,714 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
17:46:05,719 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [STDOUT]
17:46:05,724 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
17:46:05,740 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT
level set to ERROR
17:46:05,740 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could
not find Janino library on the class path. Skipping conditional processing.
17:46:05,740 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See
also http://logback.qos.ch/codes.html#ifJanino
17:46:05,740 |-ERROR in ch.qos.logback.core.joran.action.AppenderRefAction -
Could not find an AppenderAttachable at the top of execution stack. Near
[appender-ref] line 59
17:46:05,740 |-WARN in ch.qos.logback.classic.joran.action.RootLoggerAction -
The object on the top the of the stack is not the root logger
17:46:05,740 |-WARN in ch.qos.logback.classic.joran.action.RootLoggerAction -
It is: ch.qos.logback.core.joran.conditional.IfAction
17:46:05,740 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- End of configuration.
17:46:05,741 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@58e3a2c7
- Registering current configuration as safe fallback point
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-6885) CTAS for empty output doesn't create parquet file or folder
[
https://issues.apache.org/jira/browse/DRILL-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-6885.
-
Resolution: Fixed
> CTAS for empty output doesn't create parquet file or folder
> ---
>
> Key: DRILL-6885
> URL: https://issues.apache.org/jira/browse/DRILL-6885
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Priority: Major
> Fix For: 1.17.0
>
>
> CTAS for empty output can create empty tables based on the empty json or csv
> files. But it doesn't work for parquet files.
> See examples below:
> {code:java}
> 0: jdbc:drill:zk=local> use dfs.tmp;
> +---+--+
> | ok | summary|
> +---+--+
> | true | Default schema changed to [dfs.tmp] |
> +---+--+
> 1 row selected (0.087 seconds)
> 0: jdbc:drill:zk=local> select * from `empty_dir`;
> +--+
> | |
> +--+
> +--+
> No rows selected (0.083 seconds)
> 0: jdbc:drill:zk=local> alter session set `store.format` = 'json';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.079 seconds)
> 0: jdbc:drill:zk=local> create table `empty_json` as select * from
> `empty_dir`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 0 |
> +---++
> 1 row selected (0.128 seconds)
> 0: jdbc:drill:zk=local> select * from `empty_json`;
> +--+
> | |
> +--+
> +--+
> No rows selected (0.086 seconds)
> 0: jdbc:drill:zk=local> alter session set `store.format` = 'csv';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.073 seconds)
> 0: jdbc:drill:zk=local> create table `empty_csv` as select * from `empty_dir`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 0 |
> +---++
> 1 row selected (0.135 seconds)
> 0: jdbc:drill:zk=local> select * from `empty_csv`;
> +--+
> | columns |
> +--+
> | [] |
> +--+
> 1 row selected (0.086 seconds)
> 0: jdbc:drill:zk=local> alter session set `store.format` = 'parquet';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.073 seconds)
> 0: jdbc:drill:zk=local> create table `empty_parquet` as select * from
> `empty_dir`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 0 |
> +---++
> 1 row selected (0.099 seconds)
> 0: jdbc:drill:zk=local> select * from `empty_parquet`;
> 20:41:01.619 [23f692c1-8994-9fc8-2ce4-5fc6135ebcc9:foreman] ERROR
> o.a.calcite.runtime.CalciteException -
> org.apache.calcite.sql.validate.SqlValidatorException: Object 'empty_parquet'
> not found
> 20:41:01.619 [23f692c1-8994-9fc8-2ce4-5fc6135ebcc9:foreman] ERROR
> o.a.calcite.runtime.CalciteException -
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 15 to
> line 1, column 29: Object 'empty_parquet' not found
> 20:41:01.622 [Client-1] ERROR o.a.calcite.runtime.CalciteException -
> org.apache.calcite.sql.validate.SqlValidatorException: Object 'empty_parquet'
> not found
> 20:41:01.623 [Client-1] ERROR o.a.calcite.runtime.CalciteException -
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 15 to
> line 1, column 29: Object 'empty_parquet' not found: Object 'empty_parquet'
> not found
> Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 29: Object
> 'empty_parquet' not found
> [Error Id: 879730dc-aad6-4fc7-9c62-9ad8bbc99d42 on vitalii-pc:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-4949) Need better handling of empty parquet files
[
https://issues.apache.org/jira/browse/DRILL-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-4949.
-
Resolution: Fixed
> Need better handling of empty parquet files
> ---
>
> Key: DRILL-4949
> URL: https://issues.apache.org/jira/browse/DRILL-4949
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Krystal
>Priority: Major
> Fix For: 1.17.0
>
>
> I have an empty parquet file created from hive. When I tried to query
> against this table I got "IllegalArgumentException".
> {code}
> select * from `test_dir/voter_empty`;
> Error: SYSTEM ERROR: IllegalArgumentException: MinorFragmentId 0 has no read
> entries assigned
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: MinorFragmentId 0 has no read entries assigned
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (java.lang.IllegalArgumentException) MinorFragmentId 0 has no
> read entries assigned
> com.google.common.base.Preconditions.checkArgument():122
> org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan():824
> org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan():101
> org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan():68
> org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan():35
> org.apache.drill.exec.physical.base.AbstractGroupScan.accept():63
> org.apache.drill.exec.planner.fragment.Materializer.visitOp():102
> org.apache.drill.exec.planner.fragment.Materializer.visitOp():35
>
> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitProject():79
> org.apache.drill.exec.physical.config.Project.accept():51
> org.apache.drill.exec.planner.fragment.Materializer.visitStore():82
> org.apache.drill.exec.planner.fragment.Materializer.visitStore():35
>
> org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitScreen():202
> org.apache.drill.exec.physical.config.Screen.accept():98
>
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.generateWorkUnit():283
>
> org.apache.drill.exec.planner.fragment.SimpleParallelizer.getFragments():127
> org.apache.drill.exec.work.foreman.Foreman.getQueryWorkUnit():596
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan():426
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1010
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745 (state=,code=0)
> {code}
> Either drill should block the query and display a user friendly error message
> or allow the query to run and return empty result.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-1834) Misleading error message when querying an empty Parquet file
[
https://issues.apache.org/jira/browse/DRILL-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-1834.
-
Resolution: Fixed
> Misleading error message when querying an empty Parquet file
>
>
> Key: DRILL-1834
> URL: https://issues.apache.org/jira/browse/DRILL-1834
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 0.7.0
>Reporter: Aman Sinha
>Priority: Minor
> Fix For: 1.17.0
>
>
> It is possible that a CTAS may fail and create an empty Parquet file. When
> we run a query against this file, we get a misleading error message from the
> planner that hides the original IOException, although the log file does have
> the original exception:
> {code:sql}
> 0: jdbc:drill:zk=local> select count(*) from dfs.`/tmp/empty.parquet`;
> Query failed: Query failed: Unexpected exception during fragment
> initialization: Internal error: Error while applying rule
> DrillPushProjIntoScan, args
> [rel#77:ProjectRel.NONE.ANY([]).[](child=rel#76:Subset#0.ENUMERABLE.ANY([]).[],$f0=0),
> rel#68:EnumerableTableAccessRel.ENUMERABLE.ANY([]).[](table=[dfs,
> /tmp/empty.parquet])]
> {code}
> The cause of the exception is in the logs:
> Caused by: java.io.IOException: Could not read footer:
> java.lang.RuntimeException: file:/tmp/empty.parquet is not a Parquet file
> (too small)
> at
> parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:195)
> ~[parquet-hadoop-1.5.1-drill-r4.jar:0.7.0-SNAPSHOT]
> at
> parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:208)
> ~[parquet-hadoop-1.5.1-drill-r4.jar:0.7.0-SNAPSHOT]
> at
> parquet.hadoop.ParquetFileReader.readFooters(ParquetFileReader.java:224)
> ~[parquet-hadoop-1.5.1-drill-r4.jar:0.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooter(ParquetGroupScan.java:208)
> ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7393) Revisit Drill tests to ensure that patching is executed before any test run
Arina Ielchiieva created DRILL-7393:
---
Summary: Revisit Drill tests to ensure that patching is executed
before any test run
Key: DRILL-7393
URL: https://issues.apache.org/jira/browse/DRILL-7393
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Apache Drill patches some Protobuf and Guava classes (see GuavaPatcher,
ProtobufPatcher), patching should be done before classes to be patched are
loaded. That's why this operation is executed in static block in Drillbit class.
Some tests in java-exec module use Drillbit class, some extend DrillTest class,
both of them patch Guava. But there are some tests that do not call patcher but
load classes to be patched. For example,
{{org.apache.drill.exec.sql.TestSqlBracketlessSyntax}} loads Guava
Preconditions class. If such tests run before tests that require patching,
tests run will fail since patching won't be successful. Patchers code does not
fail application if patching was not complete, just logs warning
({{logger.warn("Unable to patch Guava classes.", e);}}), so sometimes it hard
to identify unit tests failure root cause.
We need to revisit all Drill tests to ensure that all of them extend common
test base class which patchers Protobuf and Guava classes in static block. Also
refactor Patcher classes to have assert so patching fails during unit testing
if there are any problems.
After all tests are revised, we can remove {{metastore-test}} execution from
main.xml in {{maven-surefire-plugin}} which was added to ensure that all
Metastore tests run in a separate JVM where patching is done in first place
since Iceberg Metastore heavily depends on patched Guava Preconditions class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (DRILL-6835) Schema Provision using File / Table Function
[ https://issues.apache.org/jira/browse/DRILL-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-6835. - Resolution: Fixed > Schema Provision using File / Table Function > > > Key: DRILL-6835 > URL: https://issues.apache.org/jira/browse/DRILL-6835 > Project: Apache Drill > Issue Type: New Feature >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.17.0 > > > Schema Provision using File / Table Function design document: > https://docs.google.com/document/d/1mp4egSbNs8jFYRbPVbm_l0Y5GjH3HnoqCmOpMTR_g4w/edit?usp=sharing > Phase 1 functional specification - > https://docs.google.com/document/d/1ExVgx2FDqxAz5GTqyWt-_1-UqwRSTGLGEYuc8gsESG8/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7368) Query from Iceberg Metastore fails if filter column contains null
Arina Ielchiieva created DRILL-7368: --- Summary: Query from Iceberg Metastore fails if filter column contains null Key: DRILL-7368 URL: https://issues.apache.org/jira/browse/DRILL-7368 Project: Apache Drill Issue Type: Bug Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 When querying data from Drill Iceberg Metastore query fails if filter column contains null. Problem is in Iceberg implementation - https://github.com/apache/incubator-iceberg/pull/443 Fix steps: upgrade to latest Iceberg commit which includes appropriate fix. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7367) Remove Server details from response headers
Arina Ielchiieva created DRILL-7367:
---
Summary: Remove Server details from response headers
Key: DRILL-7367
URL: https://issues.apache.org/jira/browse/DRILL-7367
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.17.0
Drill response headers include Server information which is considered to be a
vulnerability.
{noformat}
curl http://localhost:8047/cluster.json -v -k
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8047 (#0)
> GET /cluster.json HTTP/1.1
> Host: localhost:8047
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 05 Sep 2019 12:47:53 GMT
< Content-Type: application/json
< Content-Length: 436
< Server: Jetty(9.3.25.v20180904)
{noformat}
https://pentest-tools.com/blog/essential-http-security-headers/
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7361) Add Map (Dict) support for schema file provisioning
Arina Ielchiieva created DRILL-7361: --- Summary: Add Map (Dict) support for schema file provisioning Key: DRILL-7361 URL: https://issues.apache.org/jira/browse/DRILL-7361 Project: Apache Drill Issue Type: New Feature Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Once Dict is added to row set framework, schema commands must be able to process this type. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7360) Refactor WatchService code in Drillbit class
Arina Ielchiieva created DRILL-7360: --- Summary: Refactor WatchService code in Drillbit class Key: DRILL-7360 URL: https://issues.apache.org/jira/browse/DRILL-7360 Project: Apache Drill Issue Type: Task Affects Versions: 1.16.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Refactor WatchService to user proper code (see https://docs.oracle.com/javase/tutorial/essential/io/notification.html for details) and fix concurrency issues connected with variables assigning from different thread. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7358) Text reader returns nothing for count queries over empty files
Arina Ielchiieva created DRILL-7358:
---
Summary: Text reader returns nothing for count queries over empty
files
Key: DRILL-7358
URL: https://issues.apache.org/jira/browse/DRILL-7358
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Paul Rogers
Fix For: 1.17.0
If we do count over empty CSV files (with or without headers), there is not
result.
Though expecting result is 0.
Unit tests examples:
{code}
@Test
public void testCount() throws Exception {
String fileName = "headersOnly.csv";
try (PrintWriter out = new PrintWriter(new FileWriter(new File(testDir,
fileName {
out.print("a,b,c"); // note: no \n in the end
}
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
{code}
@Test
public void testCount() throws Exception {
String fileName = "empty.csv";
File file = new File(testDir, fileName);
assertTrue(file.createNewFile());
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7357) Expose Drill Metastore Metadata through INFORMATION_SCHEMA
Arina Ielchiieva created DRILL-7357: --- Summary: Expose Drill Metastore Metadata through INFORMATION_SCHEMA Key: DRILL-7357 URL: https://issues.apache.org/jira/browse/DRILL-7357 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Document: https://docs.google.com/document/d/10CkLdrlUJUNRrHKLeo8jTUJB8xAP1D0byTOvn8wNoF0/edit#heading=h.gzj2dj5a4yds Sections: 5.19 INFORMATION_SCHEMA updates 4.3.2 Using the statistics -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7347) Upgrade Apache Iceberg to released version
Arina Ielchiieva created DRILL-7347: --- Summary: Upgrade Apache Iceberg to released version Key: DRILL-7347 URL: https://issues.apache.org/jira/browse/DRILL-7347 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Currently Drill uses Apache Iceberg build on certain commit using JitPack since there is no official released version. Once Iceberg first version is released, we need to use officially released version instead of commit. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DRILL-6528) Planner setting the wrong number of records to read (Parquet Reader)
[ https://issues.apache.org/jira/browse/DRILL-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-6528. - Resolution: Fixed Resolved in the scope of DRILL-4517. > Planner setting the wrong number of records to read (Parquet Reader) > > > Key: DRILL-6528 > URL: https://issues.apache.org/jira/browse/DRILL-6528 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Reporter: salim achouche >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.17.0 > > > - Recently fixed the Flat Parquet reader to honor the number of records to > read > - Though few tests failed: > TestUnionDistinct.testUnionDistinctEmptySides:356 Different number of records > returned expected:<5> but was:<1> > TestUnionAll.testUnionAllEmptySides:355 Different number of records returned > expected:<5> but was:<1> > - I debugged one of them and realized the Planner was setting the wrong > number of rows to read (in this case, one) > - You can put a break point and see this happening: > Class: ParquetGroupScan > Method: updateRowGroupInfo(long maxRecords) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7339) Upgrade to Iceberg latest commits to fix issue with orphan files after delete in transaction
Arina Ielchiieva created DRILL-7339: --- Summary: Upgrade to Iceberg latest commits to fix issue with orphan files after delete in transaction Key: DRILL-7339 URL: https://issues.apache.org/jira/browse/DRILL-7339 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Drill Metastore executes many operations in transaction including delete. Currently Iceberg creates orphan files when executing delete in transaction and these files cannot be expired and keep pilling up. Iceberg issue - https://github.com/apache/incubator-iceberg/issues/330. When #330 is fixed, we need to update Iceberg commit to ensure these issue is resolved in Drill as well. PR with the fix - https://github.com/apache/incubator-iceberg/pull/352 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7335) Error when reading csv file with headers only
Arina Ielchiieva created DRILL-7335:
---
Summary: Error when reading csv file with headers only
Key: DRILL-7335
URL: https://issues.apache.org/jira/browse/DRILL-7335
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Text CSV
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
Fix For: 1.17.0
Prerequisites: file contains only headers with /n: id,name.
Error:
{noformat}
org.apache.drill.exec.rpc.RpcException:
org.apache.drill.common.exceptions.UserRemoteException: EXECUTION_ERROR ERROR:
File file:onlyHeaders.csv
Fragment 0:0
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DRILL-7334) Update Iceberg Metastore Parquet write mode
Arina Ielchiieva created DRILL-7334: --- Summary: Update Iceberg Metastore Parquet write mode Key: DRILL-7334 URL: https://issues.apache.org/jira/browse/DRILL-7334 Project: Apache Drill Issue Type: Improvement Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Initially Iceberg Parquet writer was using OVERWRITE mode by default. After f4fc8ff, default mode is CREATE. Need to update Iceberg Metastore code to comply with the latest changes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7331) Support Icebrg
Arina Ielchiieva created DRILL-7331: --- Summary: Support Icebrg Key: DRILL-7331 URL: https://issues.apache.org/jira/browse/DRILL-7331 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7330) Implement metadata usage for text format plugin
Arina Ielchiieva created DRILL-7330: --- Summary: Implement metadata usage for text format plugin Key: DRILL-7330 URL: https://issues.apache.org/jira/browse/DRILL-7330 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Volodymyr Vysotskyi -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7329) Implement metadata usage for parquet format plugin
Arina Ielchiieva created DRILL-7329: --- Summary: Implement metadata usage for parquet format plugin Key: DRILL-7329 URL: https://issues.apache.org/jira/browse/DRILL-7329 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 1. Implement retrieval steps (include auto-refresh policy, retry attempts, fallback to file metadata). 2. Change the current group scan to leverage Schema from Metastore; 3. Verify that metadata is used correctly for metastore classes Add options: planner.metadata.use_schema planner.metadata.use_statistics metadata.ctas.auto_collect metadata.fallback_to_file_metadata metastore.auto-refresh -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7314) Use TupleMetadata instead of concrete implementation
Arina Ielchiieva created DRILL-7314: --- Summary: Use TupleMetadata instead of concrete implementation Key: DRILL-7314 URL: https://issues.apache.org/jira/browse/DRILL-7314 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Use TupleMetadata instead of concrete implementation. Currently TupleSchema is used during ser / de since TupleMetadata is interface. This Jira aims to support TupleMetadata ser / de instead of concrete implementation. This would be achieved using json sub types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-7262) Parse Error appears on attempting to run several SQL queries at the same time in SQLLine
[ https://issues.apache.org/jira/browse/DRILL-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-7262. - Resolution: Fixed Resolved with 2766e653cda8b1de817b234c66b0058e707750d0 commit id. > Parse Error appears on attempting to run several SQL queries at the same time > in SQLLine > > > Key: DRILL-7262 > URL: https://issues.apache.org/jira/browse/DRILL-7262 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Pavel Semenov >Assignee: Volodymyr Vysotskyi >Priority: Minor > Fix For: 1.17.0 > > Attachments: 2.png > > > *STEPS TO REPRODUCE* > # Run SqlLine > # Submit several SQL queries at the same time, e.g. (select * from > sys.version; select * from sys.version; ) > # Observe the result > *EXPECTED RESULT* > Several query results appear > *ACTUAL RESULT* > Parse Error appears > !2.png|thumbnail! > *ADDITIONAL INFO* > Current issue will be fixed in scope of > [https://github.com/julianhyde/sqlline/pull/297] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-7158) null values for varchar, interval, boolean are displayed as empty string in SqlLine
[
https://issues.apache.org/jira/browse/DRILL-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-7158.
-
Resolution: Fixed
Resolved with 2766e653cda8b1de817b234c66b0058e707750d0 commit id.
> null values for varchar, interval, boolean are displayed as empty string in
> SqlLine
> ---
>
> Key: DRILL-7158
> URL: https://issues.apache.org/jira/browse/DRILL-7158
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.17.0
>
>
> null values for varchar, interval, boolean are displayed as empty string in
> SqlLine.
> Caused by SqlLine bug: [https://github.com/julianhyde/sqlline/issues/288]
> Possible workaround to set nullValue other case than lower: {{!set nullValue
> Null}}.
> Should be fixed in the next SqlLine upgrade (to 1.8.0) when prior fixed in
> SqlLine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-7198) Issuing a control-C in Sqlline exits the session (it does cancel the query)
[ https://issues.apache.org/jira/browse/DRILL-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-7198. - Resolution: Fixed Resolved with 2766e653cda8b1de817b234c66b0058e707750d0 commit id. > Issuing a control-C in Sqlline exits the session (it does cancel the query) > --- > > Key: DRILL-7198 > URL: https://issues.apache.org/jira/browse/DRILL-7198 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.15.0, 1.16.0 >Reporter: Aman Sinha >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > This behavior is observed both in Drill 1.15.0 and the RC1 of 1.16.0. Run a > long-running query in sqlline and cancel it using control-c. It exits the > sqlline session although it does cancel the query. Behavior is seen in both > embedded mode and distributed mode. If the query is submitted through > sqlline and cancelled from the Web UI, it does behave correctly..the session > does not get killed and subsequent queries can be submitted in the same > sqlline session. > Same query in Drill 1.14.0 works correctly and returns the column headers > while canceling the query. > Since the query can be cancelled just fine through the Web UI, I am not > considering this a blocker for 1.16. Very likely the sqlline upgrade in > 1.15.0 changed the behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7274) Introduce ANALYZE TABLE statements
Arina Ielchiieva created DRILL-7274: --- Summary: Introduce ANALYZE TABLE statements Key: DRILL-7274 URL: https://issues.apache.org/jira/browse/DRILL-7274 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Volodymyr Vysotskyi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7273) Create operator for handling metadata
Arina Ielchiieva created DRILL-7273: --- Summary: Create operator for handling metadata Key: DRILL-7273 URL: https://issues.apache.org/jira/browse/DRILL-7273 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Volodymyr Vysotskyi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7272) Implement Drill Iceberg Metastore plugin
Arina Ielchiieva created DRILL-7272: --- Summary: Implement Drill Iceberg Metastore plugin Key: DRILL-7272 URL: https://issues.apache.org/jira/browse/DRILL-7272 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7271) Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore
Arina Ielchiieva created DRILL-7271:
---
Summary: Refactor Metadata interfaces and classes to contain all
needed information for the File based Metastore
Key: DRILL-7271
URL: https://issues.apache.org/jira/browse/DRILL-7271
Project: Apache Drill
Issue Type: Sub-task
Reporter: Arina Ielchiieva
Assignee: Volodymyr Vysotskyi
1. Merge info from metadataStatistics + statisticsKinds into one holder:
Map.
2. Rename hasStatistics to hasDescriptiveStatistics
3. Remove drill-file-metastore-plugin
4. Move
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.MetadataLevel
to metadata module, rename to MetadataType and add new value: DIRECTORY.
5. Add JSON ser/de for ColumnStatistics, StatisticsHolder.
6. Add new info classes:
{noformat}
class TableInfo {
String storagePlugin;
String workspace;
String name;
String type;
String owner;
}
class MetadataInfo {
public static final String GENERAL_INFO_KEY = "GENERAL_INFO";
public static final String DEFAULT_PARTITION_KEY = "DEFAULT_PARTITION";
MetadataType type (enum);
String key;
String identifier;
}
{noformat}
7. Modify existing metadata classes:
org.apache.drill.metastore.FileTableMetadata
{noformat}
missing fields
--
storagePlugin, workspace, tableType -> will be covered by TableInfo class
metadataType, metadataKey -> will be covered by MetadataInfo class
interestingColumns
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set partitionKeys; -> Map
{noformat}
org.apache.drill.metastore.PartitionMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
partitionValues (List)
location (String) (for directory level metadata) - directory location
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Set location; -> locations
{noformat}
org.apache.drill.metastore.FileMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
path - path to file
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
private final Path location; - should contain directory to which file belongs
{noformat}
org.apache.drill.metastore.RowGroupMetadata
{noformat}
missing fields
--
storagePlugin, workspace -> will be covered by TableInfo class
metadataType, metadataKey, metadataIdentifier -> will be covered by
MetadataInfo class
fields to modify
private final Map tableStatistics;
private final Map statisticsKinds;
{noformat}
8. Remove org.apache.drill.exec package from metastore module.
9. Rename ColumnStatisticsImpl class.
10. Separate existing classes in org.apache.drill.metastore package into
sub-packages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6742) Research and investigate a way for collecting and storing table statistics in the scope of metastore integration
[ https://issues.apache.org/jira/browse/DRILL-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-6742. - Resolution: Fixed > Research and investigate a way for collecting and storing table statistics in > the scope of metastore integration > > > Key: DRILL-6742 > URL: https://issues.apache.org/jira/browse/DRILL-6742 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > > In the scope of DRILL-1328 was made a significant work for collecting and > processing table statistics. The main show stopper for this Jira is a way of > storing collected statistics. > The aim of this Jira is to investigate how this statistics may be stored in > the metastore for different table types, a way how it may be additionally > collected and received from the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-5903) Regression: Query encounters "Waited for 15000ms, but tasks for 'Fetch parquet metadata' are not complete."
[
https://issues.apache.org/jira/browse/DRILL-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva resolved DRILL-5903.
-
Resolution: Invalid
> Regression: Query encounters "Waited for 15000ms, but tasks for 'Fetch
> parquet metadata' are not complete."
> ---
>
> Key: DRILL-5903
> URL: https://issues.apache.org/jira/browse/DRILL-5903
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Storage - Parquet
>Affects Versions: 1.11.0
>Reporter: Robert Hou
>Assignee: Arina Ielchiieva
>Priority: Critical
> Attachments: 26122f83-6956-5aa8-d8de-d4808f572160.sys.drill,
> drillbit.log
>
>
> This is a query from the Functional-Baseline-100.171 run. The test is
> /root/drillAutomation/mapr/framework/resources/Functional/parquet_storage/parquet_date/mc_parquet_date/generic/mixed1_partitioned5.q.
> Query is:
> {noformat}
> select a.int_col, b.date_col from
> dfs.`/drill/testdata/parquet_date/metadata_cache/mixed/fewtypes_null_large` a
> inner join ( select date_col, int_col from
> dfs.`/drill/testdata/parquet_date/metadata_cache/mixed/fewtypes_null_large`
> where dir0 = '1.2' and date_col > '1996-03-07' ) b on cast(a.date_col as
> date)= date_add(b.date_col, 5) where a.int_col = 7 and a.dir0='1.9' group by
> a.int_col, b.date_col
> {noformat}
> From drillbit.log:
> {noformat}
> fc65-d430-ac1103638113: SELECT SUM(col_int) OVER() sum_int FROM
> vwOnParq_wCst_35
> 2017-10-23 11:20:50,122 [26122f83-6956-5aa8-d8de-d4808f572160:foreman] ERROR
> o.a.d.exec.store.parquet.Metadata - Waited for 15000ms, but tasks for 'Fetch
> parquet metadata' are not complete. Total runnable size 3, parallelism 3.
> 2017-10-23 11:20:50,127 [26122f83-6956-5aa8-d8de-d4808f572160:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - User Error Occurred: Waited for 15000ms,
> but tasks for 'Fetch parquet metadata' are not complete. Total runnable size
> 3, parallelism 3.
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Waited for
> 15000ms, but tasks for 'Fetch parquet metadata' are not complete. Total
> runnable size 3, parallelism 3.
> [Error Id: 7484e127-ea41-4797-83c0-6619ea9b2bcd ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
> ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:151)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.Metadata.getParquetFileMetadata_v3(Metadata.java:341)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:318)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:142)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.init(ParquetGroupScan.java:934)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:227)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:190)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:170)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:66)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:144)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:100)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:85)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch(DrillPushProjIntoScan.java:62)
> [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
> [calcite-core-1.4.0-drill-r22.jar:1.4.0-drill-r22]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:811)
> [calcite-core-1.4.0-drill-r22.jar:1.4.0-drill-r22]
> at
>
[jira] [Created] (DRILL-7270) Fix non-https dependency urls
Arina Ielchiieva created DRILL-7270: --- Summary: Fix non-https dependency urls Key: DRILL-7270 URL: https://issues.apache.org/jira/browse/DRILL-7270 Project: Apache Drill Issue Type: Task Affects Versions: 1.16.0 Reporter: Arina Ielchiieva Assignee: Dmytriy Grinchenko Fix For: 1.17.0 Review any build scripts and configurations for insecure urls and make appropriate fixes to use secure urls. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7236) SqlLine 1.8 upgrade
Arina Ielchiieva created DRILL-7236: --- Summary: SqlLine 1.8 upgrade Key: DRILL-7236 URL: https://issues.apache.org/jira/browse/DRILL-7236 Project: Apache Drill Issue Type: Task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva SqlLine 1.8 upgrade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7205) Drill fails to start when authentication is disabled
Arina Ielchiieva created DRILL-7205:
---
Summary: Drill fails to start when authentication is disabled
Key: DRILL-7205
URL: https://issues.apache.org/jira/browse/DRILL-7205
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0, 1.16.0
Reporter: Arina Ielchiieva
Fix For: Future
drill-override.conf content:
{noformat}
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
impersonation: {
enabled: true,
max_chained_user_hops: 3
},
security.user.auth: {
enabled: false,
packages += "org.apache.drill.exec.rpc.user.security",
impl: "pam",
pam_profiles: [ "sudo", "login" ]
}
}
{noformat}
Note that authentication is disabled {{enabled: false}}.
Drill fails during start up:
{noformat}
DRILL_ARGS - " -u jdbc:drill:zk=local"
Calculating HADOOP_CLASSPATH ...
HBASE_HOME not detected...
Calculating Drill classpath...
Error: Failure in starting embedded Drillbit:
org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
the native library of JPAM (Pluggable Authenticator Module API). Make sure to
set Drillbit JVM option 'java.library.path' to point to the directory where the
native JPAM exists.:no jpam in java.library.path (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit:
org.apache.drill.exec.exception.DrillbitStartupException: Problem in finding
the native library of JPAM (Pluggable Authenticator Module API). Make sure to
set Drillbit JVM option 'java.library.path' to point to the directory where the
native JPAM exists.:no jpam in java.library.path
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:143)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
at sqlline.Commands.connect(Commands.java:1278)
at sqlline.Commands.connect(Commands.java:1172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:736)
at sqlline.SqlLine.initArgs(SqlLine.java:428)
at sqlline.SqlLine.begin(SqlLine.java:531)
at sqlline.SqlLine.start(SqlLine.java:270)
at sqlline.SqlLine.main(SqlLine.java:201)
Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Problem in
finding the native library of JPAM (Pluggable Authenticator Module API). Make
sure to set Drillbit JVM option 'java.library.path' to point to the directory
where the native JPAM exists.:no jpam in java.library.path
at
org.apache.drill.exec.rpc.user.security.PamUserAuthenticator.setup(PamUserAuthenticator.java:52)
at
org.apache.drill.exec.rpc.user.security.UserAuthenticatorFactory.createAuthenticator(UserAuthenticatorFactory.java:98)
at
org.apache.drill.exec.rpc.security.AuthenticatorProviderImpl.(AuthenticatorProviderImpl.java:66)
at
org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:83)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:161)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:125)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:133)
... 18 more
Caused by: java.lang.UnsatisfiedLinkError: no jpam in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at net.sf.jpam.Pam.(Pam.java:51)
at
org.apache.drill.exec.rpc.user.security.PamUserAuthenticator.setup(PamUserAuthenticator.java:46)
... 24 more
Apache Drill 1.16.0
"Let's Drill something more solid than concrete."
apache drill> select * from sys.version;
No current connection
{noformat}
If we remove authentication block from config, Drill starts without errors:
{noformat}
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
impersonation: {
enabled: true,
max_chained_user_hops: 3
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7204) Add proper validation when creating plugin
Arina Ielchiieva created DRILL-7204: --- Summary: Add proper validation when creating plugin Key: DRILL-7204 URL: https://issues.apache.org/jira/browse/DRILL-7204 Project: Apache Drill Issue Type: Bug Affects Versions: 1.16.0 Reporter: Arina Ielchiieva Assignee: Anton Gozhiy Fix For: 1.17.0 1. Currently there is no failure when user attempts to create plugin without name. Screenshot attached. I think we need proper plugin name validation when creating plugin. 2. When disabling and deleting plugin, alerts are used. Its better to use more user friendly message window. Screenshot attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
