[jira] [Commented] (DRILL-4309) Make this option store.hive.optimize_scan_with_native_readers=true default
[ https://issues.apache.org/jira/browse/DRILL-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311498#comment-15311498 ] ASF GitHub Bot commented on DRILL-4309: --- Github user adeneche commented on the issue: https://github.com/apache/drill/pull/350 +1, LGTM > Make this option store.hive.optimize_scan_with_native_readers=true default > -- > > Key: DRILL-4309 > URL: https://issues.apache.org/jira/browse/DRILL-4309 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning & Optimization >Reporter: Sean Hsuan-Yi Chu >Assignee: Sean Hsuan-Yi Chu > Fix For: 1.7.0 > > > This new feature has been around and used/tests in many scenarios. > We should enable this feature by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4571) Add link to local Drill logs from the web UI
[ https://issues.apache.org/jira/browse/DRILL-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311317#comment-15311317 ] Krystal commented on DRILL-4571: I installed the "DRILL-4571-fix" branch. Each entry in drillbit_queries.json are now being displayed on its own line from Chrome browser. Downloading of log files now saved with correct name. And lastly, the UI shows correct last 10,000 lines of a log file. > Add link to local Drill logs from the web UI > > > Key: DRILL-4571 > URL: https://issues.apache.org/jira/browse/DRILL-4571 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting > Fix For: 1.7.0 > > Attachments: display_log.JPG, drillbit_download.log.gz, > drillbit_queries_json_screenshot.jpg, drillbit_ui.log, log_list.JPG > > > Now we have link to the profile from the web UI. > It will be handy for the users to have the link to local logs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4703) Use same root allocator across same JDBC connections on the client side
Krystal created DRILL-4703: -- Summary: Use same root allocator across same JDBC connections on the client side Key: DRILL-4703 URL: https://issues.apache.org/jira/browse/DRILL-4703 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Reporter: Krystal Each jdbc drill client in the same JVM spawns its own root allocator instead of using only one. This can lead to incorrect accounting. For example, the drill test framework uses connection pools which exposes this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4699) Add Description Column in sys.options
[ https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zelaine Fong updated DRILL-4699: Assignee: Sudheesh Katkam > Add Description Column in sys.options > - > > Key: DRILL-4699 > URL: https://issues.apache.org/jira/browse/DRILL-4699 > Project: Apache Drill > Issue Type: Improvement > Components: Server, Documentation >Affects Versions: 1.6.0 >Reporter: John Omernik >Assignee: Sudheesh Katkam > > select * from sys.options provides a user with a strong understanding of what > options are available to Drill. These options are not well documented. Some > options are "experimental" other options have a function only in specific > cases (writers vs readers for example). If we had a large text field for > description, we could enforce documentation of the settings are option > creation time, and the description of the setting could change as the > versions change (i.e. when an option graduates to being supported from being > experimental, it would be changed in the version the user is using. I.e. when > they run select * from sys.options, they know the exact state of the option > every time they query. It could also facilitate better self documentation via > QA on pull requests "Did you update the sys.options.desc?" This makes it > easier for users, and admins in the use of Drill in an enterprise. > The first step is adding the field, and then going back and filling in the > desc for each option. (Another JIRA after the option is available) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4041) Random "Buffer has negative reference count" error
[ https://issues.apache.org/jira/browse/DRILL-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-4041: --- Summary: Random "Buffer has negative reference count" error (was: Parquet library update causing random "Buffer has negative reference count") > Random "Buffer has negative reference count" error > -- > > Key: DRILL-4041 > URL: https://issues.apache.org/jira/browse/DRILL-4041 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.3.0 >Reporter: Rahul Challapalli >Assignee: Steven Phillips >Priority: Critical > > git commit # 39582bd60c9e9b16aba4f099d434e927e7e5 > After the parquet library update commit, we started seeing the below error > randomly causing failures in the Extended Functional Suite. > {code} > Failed with exception > java.lang.IllegalArgumentException: Buffer has negative reference count. > at > oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:250) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:239) > at > oadd.org.apache.drill.exec.vector.BaseDataValueVector.clear(BaseDataValueVector.java:39) > at > oadd.org.apache.drill.exec.vector.NullableIntVector.clear(NullableIntVector.java:150) > at > oadd.org.apache.drill.exec.record.SimpleVectorWrapper.clear(SimpleVectorWrapper.java:84) > at > oadd.org.apache.drill.exec.record.VectorContainer.zeroVectors(VectorContainer.java:312) > at > oadd.org.apache.drill.exec.record.VectorContainer.clear(VectorContainer.java:296) > at > oadd.org.apache.drill.exec.record.RecordBatchLoader.clear(RecordBatchLoader.java:183) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.cleanup(DrillResultSetImpl.java:139) > at org.apache.drill.jdbc.impl.DrillCursor.close(DrillCursor.java:333) > at > oadd.net.hydromatic.avatica.AvaticaResultSet.close(AvaticaResultSet.java:110) > at > org.apache.drill.jdbc.impl.DrillResultSetImpl.close(DrillResultSetImpl.java:169) > at > org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:233) > at > org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:89) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4041) Parquet library update causing random "Buffer has negative reference count"
[ https://issues.apache.org/jira/browse/DRILL-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310942#comment-15310942 ] Krystal commented on DRILL-4041: git.commit.id.abbrev=6ddd5fa Ran into this error while running the tpcds advanced test with concurrency of 20 and 2 drillbits running. bin/run_tests -s Advanced/tpcds/tpcds_sf100/original -g smoke -n 20 Have not been able to reproduce the error again. {code} Exiting due to uncaught exception java.util.concurrent.ExecutionException: java.lang.IllegalStateException: DrillBuf[1937] refCnt has gone negative. Buffer Info: ledger[1128] allocator: ROOT), isOwning: false, size: 4, references: -1, life: 2852193038058198..2852195352841417, allocatorManager: [1128, life: 2852193036883482..2852195352867362] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.drill.test.framework.CancelingExecutor$1.run(CancelingExecutor.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: DrillBuf[1937] refCnt has gone negative. Buffer Info: ledger[1128] allocator: ROOT), isOwning: false, size: 4, references: -1, life: 2852193038058198..2852195352841417, allocatorManager: [1128, life: 2852193036883482..2852195352867362] at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:269) at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:244) at oadd.org.apache.drill.exec.vector.BaseDataValueVector.clear(BaseDataValueVector.java:39) at oadd.org.apache.drill.exec.vector.VarCharVector.clear(VarCharVector.java:197) at oadd.org.apache.drill.exec.vector.NullableVarCharVector.clear(NullableVarCharVector.java:142) at oadd.org.apache.drill.exec.record.SimpleVectorWrapper.clear(SimpleVectorWrapper.java:73) at oadd.org.apache.drill.exec.record.VectorContainer.zeroVectors(VectorContainer.java:365) at oadd.org.apache.drill.exec.record.VectorContainer.clear(VectorContainer.java:336) at oadd.org.apache.drill.exec.record.RecordBatchLoader.clear(RecordBatchLoader.java:215) at org.apache.drill.jdbc.impl.DrillResultSetImpl.cleanup(DrillResultSetImpl.java:148) at org.apache.drill.jdbc.impl.DrillResultSetImpl.cancel(DrillResultSetImpl.java:139) at oadd.net.hydromatic.avatica.AvaticaStatement.cancel(AvaticaStatement.java:177) at org.apache.drill.jdbc.impl.DrillStatementImpl.cancel(DrillStatementImpl.java:259) at org.apache.drill.test.framework.DrillTestJdbc.cancel(DrillTestJdbc.java:311) at org.apache.drill.test.framework.CancelingExecutor$1$1.run(CancelingExecutor.java:74) ... 5 more {code} > Parquet library update causing random "Buffer has negative reference count" > --- > > Key: DRILL-4041 > URL: https://issues.apache.org/jira/browse/DRILL-4041 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.3.0 >Reporter: Rahul Challapalli >Assignee: Steven Phillips >Priority: Critical > > git commit # 39582bd60c9e9b16aba4f099d434e927e7e5 > After the parquet library update commit, we started seeing the below error > randomly causing failures in the Extended Functional Suite. > {code} > Failed with exception > java.lang.IllegalArgumentException: Buffer has negative reference count. > at > oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:250) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:259) > at oadd.io.netty.buffer.DrillBuf.release(DrillBuf.java:239) > at > oadd.org.apache.drill.exec.vector.BaseDataValueVector.clear(BaseDataValueVector.java:39) > at > oadd.org.apache.drill.exec.vector.NullableIntVector.clear(NullableIntVector.java:150) > at > oadd.org.apache.drill.exec.record.SimpleVectorWrapper.clear(SimpleVectorWrapper.java:84) > at > oadd.org.apache.drill.exec.record.VectorContainer.zeroVectors(VectorContainer.java:312) > at > oadd.org.apache.drill.exec.record.VectorContainer.clear(VectorContainer.java:296) > at >
[jira] [Commented] (DRILL-2593) 500 error when crc for a query profile is out of sync
[ https://issues.apache.org/jira/browse/DRILL-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310569#comment-15310569 ] John Omernik commented on DRILL-2593: - Big +1 one from me, here is my user list post on the subject. This should be easily reproducible, and for end users/admins it could be quite the issue trying the track down the troublesome profile. We should really address this for user exp reasons. John Post from User list: It is absolutely a bad profile, however, it was not a empty file profile, instead, it was a profile where the write didn't complete (likely due to my bad cluster state post). I was able to reproduce it synthetically like this, say you had a profile that ended similar to this: ..."endpoint":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":1464730828126,"lastProgress":1464730828126}]}],"user":"adm"} (The ... represents the part I removed for brevity in this email, the rest of the profile is below) We can reproduce the UI profiles issue by removing ,"user":"adm"} so it ends like this: ..."endpoint":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":1464730828126,"lastProgress":1464730828126}]}] Let me know if you know of a JIRA, otherwise I'll open one, this is a pain for users, because you have to go through an validate the json on every profile to find the one that is breaking your UI. John Full "Working" Profile: {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,"start":1464730828036,"end":1464730828131,"query":"alter session set `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"state":3,"minorFragmentId":0,"operatorProfile":[{"inputProfile":[{"records":1,"batches":1,"schemas":1}],"operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"peakLocalMemoryAllocated":4530432,"waitNanos":0},{"inputProfile":[{"records":1,"batches":1,"schemas":1}],"operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"peakLocalMemoryAllocated":4530176,"metric":[{"metricId":0,"longValue":59}],"waitNanos":1936593}],"startTime":1464730828114,"endTime":1464730828123,"memoryUsed":0,"maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":1464730828126,"lastProgress":1464730828126}]}],"user":"adm"} Full "Broken" Profile: {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,"start":1464730828036,"end":1464730828131,"query":"alter session set `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"state":3,"minorFragmentId":0,"operatorProfile":[{"inputProfile":[{"records":1,"batches":1,"schemas":1}],"operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"peakLocalMemoryAllocated":4530432,"waitNanos":0},{"inputProfile":[{"records":1,"batches":1,"schemas":1}],"operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"peakLocalMemoryAllocated":4530176,"metric":[{"metricId":0,"longValue":59}],"waitNanos":1936593}],"startTime":1464730828114,"endTime":1464730828123,"memoryUsed":0,"maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":1464730828126,"lastProgress":1464730828126}]}] > 500 error when crc for a query profile is out of sync > - > > Key: DRILL-2593 > URL: https://issues.apache.org/jira/browse/DRILL-2593 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 0.7.0 >Reporter: Jason Altekruse > Fix For: Future > > > To reproduce, on a machine where an embedded drillbit has been run, edit one > of the profiles stored in /tmp/drill/profiles and try to navigate to the > profiles page on the Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4702) MBean drill.queries.enqueued is not collecting enqueued queries
[ https://issues.apache.org/jira/browse/DRILL-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zelaine Fong updated DRILL-4702: Assignee: Sudheesh Katkam > MBean drill.queries.enqueued is not collecting enqueued queries > > > Key: DRILL-4702 > URL: https://issues.apache.org/jira/browse/DRILL-4702 > Project: Apache Drill > Issue Type: Bug > Components: Server >Reporter: Krystal >Assignee: Sudheesh Katkam > > The metric drill.queries.enqueued suppose to collect info on the number of > queries submitted but have not started yet. However currently the enqueued > queries are being counted as part of the running queries and not part of the > enqueued queries. In the UI Profile page, the state of the waiting queries > is "enqueued". If the drill.queries.enqueued metric is meant for something > else, then the name should be change as it is not matching what the UI > displays. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4702) MBean drill.queries.enqueued is not collecting enqueued queries
Krystal created DRILL-4702: -- Summary: MBean drill.queries.enqueued is not collecting enqueued queries Key: DRILL-4702 URL: https://issues.apache.org/jira/browse/DRILL-4702 Project: Apache Drill Issue Type: Bug Components: Server Reporter: Krystal The metric drill.queries.enqueued suppose to collect info on the number of queries submitted but have not started yet. However currently the enqueued queries are being counted as part of the running queries and not part of the enqueued queries. In the UI Profile page, the state of the waiting queries is "enqueued". If the drill.queries.enqueued metric is meant for something else, then the name should be change as it is not matching what the UI displays. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4573) Zero copy LIKE, REGEXP_MATCHES, SUBSTR
[ https://issues.apache.org/jira/browse/DRILL-4573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310358#comment-15310358 ] jean-claude commented on DRILL-4573: ok I have created a pull request with what I have. > Zero copy LIKE, REGEXP_MATCHES, SUBSTR > -- > > Key: DRILL-4573 > URL: https://issues.apache.org/jira/browse/DRILL-4573 > Project: Apache Drill > Issue Type: Improvement >Reporter: jean-claude >Priority: Critical > Fix For: 1.7.0 > > Attachments: DRILL-4573-3.patch.txt, DRILL-4573.patch.txt > > > All the functions using the java.util.regex.Matcher are currently creating > Java string objects to pass into the matcher.reset(). > However this creates unnecessary copy of the bytes and a Java string object. > The matcher uses a CharSequence, so instead of making a copy we can create an > adapter from the DrillBuffer to the CharSequence interface. > Gains of 25% in execution speed are possible when going over VARCHAR of 36 > chars. The gain will be proportional to the size of the VARCHAR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4701) Fix log name and missing lines in logs on Web UI
[ https://issues.apache.org/jira/browse/DRILL-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310149#comment-15310149 ] ASF GitHub Bot commented on DRILL-4701: --- GitHub user arina-ielchiieva opened a pull request: https://github.com/apache/drill/pull/511 DRILL-4701: Fix log name and missing lines in logs on Web UI You can merge this pull request into a Git repository by running: $ git pull https://github.com/arina-ielchiieva/drill DRILL-4701 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #511 commit 75b33b31bfd32b823d89e4c176bcebc009c471c1 Author: Arina IelchiievaDate: 2016-06-01T11:16:31Z DRILL-4701: Fix log name and missing lines in logs on Web UI > Fix log name and missing lines in logs on Web UI > > > Key: DRILL-4701 > URL: https://issues.apache.org/jira/browse/DRILL-4701 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: 1.7.0 > > > 1. When the log files are downloaded from the ui, the name of the downloaded > file is "download". We should save the file with the same name as the log > file (ie. drillbit.log) > 2. The last N lines of the log file displayed in the web UI do not match the > log file itself. Some lines are missing compared with actual log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4571) Add link to local Drill logs from the web UI
[ https://issues.apache.org/jira/browse/DRILL-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309917#comment-15309917 ] Arina Ielchiieva commented on DRILL-4571: - Created Jira DRILL-4701 to address changes made to fix point 1 and 3. > Add link to local Drill logs from the web UI > > > Key: DRILL-4571 > URL: https://issues.apache.org/jira/browse/DRILL-4571 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting > Fix For: 1.7.0 > > Attachments: display_log.JPG, drillbit_download.log.gz, > drillbit_queries_json_screenshot.jpg, drillbit_ui.log, log_list.JPG > > > Now we have link to the profile from the web UI. > It will be handy for the users to have the link to local logs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4701) Fix log name and missing lines in logs on Web UI
Arina Ielchiieva created DRILL-4701: --- Summary: Fix log name and missing lines in logs on Web UI Key: DRILL-4701 URL: https://issues.apache.org/jira/browse/DRILL-4701 Project: Apache Drill Issue Type: Bug Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.7.0 1. When the log files are downloaded from the ui, the name of the downloaded file is "download". We should save the file with the same name as the log file (ie. drillbit.log) 2. The last N lines of the log file displayed in the web UI do not match the log file itself. Some lines are missing compared with actual log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4199) Add Support for HBase 1.X
[ https://issues.apache.org/jira/browse/DRILL-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309343#comment-15309343 ] ASF GitHub Bot commented on DRILL-4199: --- Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/443 Hahaha! Well at least I'm consistent :P > Add Support for HBase 1.X > - > > Key: DRILL-4199 > URL: https://issues.apache.org/jira/browse/DRILL-4199 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - HBase >Affects Versions: 1.7.0 >Reporter: Divjot singh >Assignee: Aditya Kishore > > Is there any Road map to upgrade the Hbase version to 1.x series. Currently > drill supports Hbase 0.98 version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)