I completely agree with John. I think we need to make error/warning messages more friendly moving forward with any new features that ship.
Please share the JIRA that you create. But a holistic approach scares me. How would we prioritize the ones that would impact most users? Any thoughts on that. Saurabh On Thu, Jul 27, 2017 at 6:43 AM, John Omernik <[email protected]> wrote: > Hey Dan, I created the master issue for tracking improvements to error > messages here: https://issues.apache.org/jira/browse/DRILL-5692 > > If you want to create your individual error and link to it in your message, > we'll see how it works. I like the idea of a wholistic effort to improve > error messages! > > John > > > > On Thu, Jul 27, 2017 at 8:25 AM, John Omernik <[email protected]> wrote: > > > I want to bump this up. I've had a number of troubleshooting times where > > getting more concise error messages would really help me deal with my > data. > > It looks like Dan found verbose mode, but sometimes verbose isn't what we > > need, but concise. Hey Dan, maybe we could come up with a Jira that is is > > the "tracker" JIRA for error messages that are not helpful. Then each > time > > someone comes up with an error message that is frustrating or unhelpful, > we > > could create an individual JIRA for that message and link it to the > master > > JIRA. That would allow us to find, throughout the code any error messages > > that are returning with limited information. > > > > My thought is this, in the Master JIRA (which I will be happy to create) > I > > will explain the purpose of the Master JIRA, and how best to describe > error > > messages that don't help the analyst (what information to include, > showing > > the error with both normal and verbose mode, if the problem is resolved, > > what resolved the error etc). > > > > I think we should focus on errors, not issues. What I mean by that, is > > there is an error like "Null Pointer Exception" which is a horrible error > > in Java that is hard for a user to make heads or tails over, we shouldn't > > open an "bad error message" for each Null Pointer Exception issue that > > occurs. Instead, perhaps we could open one for the error "Null Pointer > > Exception" (regardless of what caused it) and provide details how we can > > better trap that and communicate it to the user... does that make sense? > > > > I will start working on a master JIRA for this. > > > > > > > > On Thu, Jul 27, 2017 at 8:19 AM, Dan Holmes < > [email protected]> > > wrote: > > > >> With verbose mode on the following is the error message. I guess this > >> confirms it is a date conversion but there isn't any help as to which > file > >> or which expression (in this statement there is only one TO_DATE > function > >> so it is obvious). > >> > >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > >> IllegalArgumentException: Invalid format: " " Fragment 1:0 [Error Id: > >> 22129e51-acbd-41f2-84f5-721541d9843b on ubuntu:31010] > >> (java.lang.IllegalArgumentException) Invalid format: " " > >> org.joda.time.format.DateTimeFormatter.parseDateTime():899 > >> org.joda.time.DateMidnight.parse():152 org.apache.drill.exec.test.gen > >> erated.ProjectorGen367.doEval():76 org.apache.drill.exec.test.gen > >> erated.ProjectorGen367.projectRecords():67 > org.apache.drill.exec.physical > >> .impl.project.ProjectRecordBatch.doWork():199 > >> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 > >> org.apache.drill.exec.physical.impl.project. > ProjectRecordBatch.innerNext():135 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():119 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():109 > >> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > >> org.apache.drill.exec.physical.impl.project. > ProjectRecordBatch.innerNext():135 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():119 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():109 > >> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext():91 > >> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >> org.apache.drill.exec.physical.impl.BaseRootExec.next():104 > >> org.apache.drill.exec.physical.impl.SingleSenderCreator$Sing > >> leSenderRootExec.innerNext():92 org.apache.drill.exec. > physical.impl.BaseRootExec.next():94 > >> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232 > >> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226 > >> java.security.AccessController.doPrivileged():-2 > >> javax.security.auth.Subject.doAs():422 org.apache.hadoop.security. > UserGroupInformation.doAs():1657 > >> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226 > >> org.apache.drill.common.SelfCleaningRunnable.run():38 > >> java.util.concurrent.ThreadPoolExecutor.runWorker():1149 > >> java.util.concurrent.ThreadPoolExecutor$Worker.run():624 > >> java.lang.Thread.run():748 > >> > >> Dan Holmes | Revenue Analytics, Inc. > >> Direct: 770.859.1255 > >> www.revenueanalytics.com > >> > >> -----Original Message----- > >> From: Dan Holmes [mailto:[email protected]] > >> Sent: Thursday, July 27, 2017 9:11 AM > >> To: [email protected] > >> Subject: How much more information can error messages provide? > >> > >> I am querying a 6GB data set in a directory. There are 34 files in the > >> directory. Here is the statement. > >> > >> CREATE TABLE dfs.root.`twm/sales_p` AS > >> SELECT to_date(columns[0], 'MM/dd/yy') transaction_date , > CAST(columns[1] > >> as INT) store_key , CAST(columns[2] AS BIGINT) item_code , columns[3] > >> position_name , CAST(CASE WHEN columns[4] = 'NULL' THEN 0 ELSE > columns[4] > >> END as INT) quantity_per_sku , CAST(columns[5] as INT) sales_quantity, > >> CAST(columns[6] as float) sales_dollars, columns[7] channel FROM > >> dfs.root.`sales` > >> > >> I get the following error. I was able to determine that columns[2] > >> needed to be a BIGINT based on an error and that columns[4] sometimes > had > >> the word "NULL" in it. In the case of NULL I had to run a statement > across > >> every column I was converting to INT (col1 = NULL or col2 = NULL ...) to > >> figure out which one it was. Now I have a similar problem with the > >> TO_DATE() function. > >> > >> Can the error message include the filename and record number? Or at > >> least the value of what it found and which expression failed? Those > kinds > >> of clues would go a long way towards making this type of detective work > >> significantly faster. The larger the dataset the faster it would be. > >> > >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > >> IllegalArgumentException: Invalid format: " " Fragment 1:0 [Error Id: > >> 3557e893-e848-4503-bf6a-73110fbd6207 on ubuntu:31010] > >> > >> v1.10 > >> > >> Dan Holmes | Architect | Revenue Analytics, Inc. > >> 300 Galleria Parkway, Suite 1900 | Atlanta, Georgia 30339 > >> Direct: 770.859.1255 <(770)%20859-1255> Cell: 404.617.3444 > >> <(404)%20617-3444> > >> www.revenueanalytics.com<https://webmail.revenueanalytics.co > >> m/owa/redir.aspx?SURL=RqmyOJRm3r383jV2nPQLyg9BvjWZqMX4- > >> tL3BHj81WfaslMWau_SCGgAdAB0AHAAOgAvAC8AdwB3AHcALgByAGUAdgBlA > >> G4AdQBlAGEAbgBhAGwAeQB0AGkAYwBzAC4AYwBvAG0A&URL=http%3a%2f% > >> 2fwww.revenueanalytics.com> > >> LinkedIn<https://webmail.revenueanalytics.com/owa/redir. > >> aspx?SURL=SrcaeiXxVTCDhl49ibCO7CHhTsNynunc_8gSjHDaikXaslMWau_ > SCGgAdAB0AHA > >> AcwA6AC8ALwB3AHcAdwAuAGwAaQBuAGsAZQBkAGkAbgAuAGMAbwBtAC8AYwB > >> vAG0AcABhAG4AeQAvAHIAZQB2AGUAbgB1AGUALQBhAG4AYQBsAHkAdABpAGM > >> AcwAtAGkAbgBjAC0A&URL=https%3a%2f%2fwww.linkedin.com% > >> 2fcompany%2frevenue-analytics-inc-> | Twitter<https://webmail.revenu > >> eanalytics.com/owa/redir.aspx?SURL=cdePsMV8TCGx8O_Rugbj-maE9 > >> C9DVT373vSJwbUc23faslMWau_SCGgAdAB0AHAAcwA6AC8ALwB0AHcAaQB0A > >> HQAZQByAC4AYwBvAG0ALwBSAGUAdgBfAEEAbgBhAGwAeQB0AGkAYwBzAA..& > >> URL=https%3a%2f%2ftwitter.com%2fRev_Analytics> > >> > >> > > >
