That's good to know Paul on the UserException. Even then, there may be exceptions sent to the user that may not prove as helpful, and I think we should strive to improve those. With the Class UserException, would it be reasonable to have a documentation link of some sort? Perhaps not a hard URL (because keeping that in sync can be hard) but if a specific UserException has been defined, doesn't it make sense to also have a page added to the documentation site with some more verbose remedies?
Saurabh - I think the nice thing about using a master jira with each specific instance of a bad error message having it's own specific issue is we CAN fix them in a way where we prioritize the issues that are most impactful, while leaving the master JIRA open as an aggregator. Say a developer wants to work on bad error messages, they can use the Master JIRA to find the individual issues they want to work on. They don't close the master JIRA, just the individual ones. It's holistic AND incremental... I guess ? :) On Thu, Jul 27, 2017 at 7:23 PM, Paul Rogers <[email protected]> wrote: > Hi All, > > As it turns out, many places in the code already use a class called > UserException. (Read that as “an exception to send to the user” rather than > “an exception that the user caused.”) Each use of UserException includes a > message and context. So, the good news is, many places in the code already > provide a useful message. > > Unfortunately, many places in the code do not provide such messages. Still > we can certainly track those that are missing and add them incrementally. > That is a pretty low cost, low risk change if we do them gradually. > > The question then becomes, why isn’t this information passed along to the > user as a nice, simple text message? That would be worth investigating. > > Other SQL engines use standard SqlCode and SqlWarning codes. A bit old > school, but allows tools to understand the meaning of the error without > parsing a vendor-specific message. We might want to add that. > > So, for priority, again from a code perspective: > > 1) Better display the messages we already have. > 2) Write a stack trace to the log, but not to the user’s console (in > Sqlline, etc.) > 3) As part of fixes and code reviews, look for spots without such messages > and add them. > > Are there better, simpler ways we could solve this? > > - Paul > > > > On Jul 27, 2017, at 3:55 PM, Saurabh Mahapatra < > [email protected]> wrote: > > > > I completely agree with John. I think we need to make error/warning > > messages more friendly moving forward with any new features that ship. > > > > Please share the JIRA that you create. But a holistic approach scares me. > > How would we prioritize the ones that would impact most users? Any > thoughts > > on that. > > > > Saurabh > > > > On Thu, Jul 27, 2017 at 6:43 AM, John Omernik <[email protected]> wrote: > > > >> Hey Dan, I created the master issue for tracking improvements to error > >> messages here: https://issues.apache.org/jira/browse/DRILL-5692 > >> > >> If you want to create your individual error and link to it in your > message, > >> we'll see how it works. I like the idea of a wholistic effort to > improve > >> error messages! > >> > >> John > >> > >> > >> > >> On Thu, Jul 27, 2017 at 8:25 AM, John Omernik <[email protected]> wrote: > >> > >>> I want to bump this up. I've had a number of troubleshooting times > where > >>> getting more concise error messages would really help me deal with my > >> data. > >>> It looks like Dan found verbose mode, but sometimes verbose isn't what > we > >>> need, but concise. Hey Dan, maybe we could come up with a Jira that is > is > >>> the "tracker" JIRA for error messages that are not helpful. Then each > >> time > >>> someone comes up with an error message that is frustrating or > unhelpful, > >> we > >>> could create an individual JIRA for that message and link it to the > >> master > >>> JIRA. That would allow us to find, throughout the code any error > messages > >>> that are returning with limited information. > >>> > >>> My thought is this, in the Master JIRA (which I will be happy to > create) > >> I > >>> will explain the purpose of the Master JIRA, and how best to describe > >> error > >>> messages that don't help the analyst (what information to include, > >> showing > >>> the error with both normal and verbose mode, if the problem is > resolved, > >>> what resolved the error etc). > >>> > >>> I think we should focus on errors, not issues. What I mean by that, is > >>> there is an error like "Null Pointer Exception" which is a horrible > error > >>> in Java that is hard for a user to make heads or tails over, we > shouldn't > >>> open an "bad error message" for each Null Pointer Exception issue that > >>> occurs. Instead, perhaps we could open one for the error "Null Pointer > >>> Exception" (regardless of what caused it) and provide details how we > can > >>> better trap that and communicate it to the user... does that make > sense? > >>> > >>> I will start working on a master JIRA for this. > >>> > >>> > >>> > >>> On Thu, Jul 27, 2017 at 8:19 AM, Dan Holmes < > >> [email protected]> > >>> wrote: > >>> > >>>> With verbose mode on the following is the error message. I guess this > >>>> confirms it is a date conversion but there isn't any help as to which > >> file > >>>> or which expression (in this statement there is only one TO_DATE > >> function > >>>> so it is obvious). > >>>> > >>>> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > >>>> IllegalArgumentException: Invalid format: " " Fragment 1:0 [Error Id: > >>>> 22129e51-acbd-41f2-84f5-721541d9843b on ubuntu:31010] > >>>> (java.lang.IllegalArgumentException) Invalid format: " " > >>>> org.joda.time.format.DateTimeFormatter.parseDateTime():899 > >>>> org.joda.time.DateMidnight.parse():152 org.apache.drill.exec.test.gen > >>>> erated.ProjectorGen367.doEval():76 org.apache.drill.exec.test.gen > >>>> erated.ProjectorGen367.projectRecords():67 > >> org.apache.drill.exec.physical > >>>> .impl.project.ProjectRecordBatch.doWork():199 > >>>> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93 > >>>> org.apache.drill.exec.physical.impl.project. > >> ProjectRecordBatch.innerNext():135 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():119 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():109 > >>>> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > >>>> org.apache.drill.exec.physical.impl.project. > >> ProjectRecordBatch.innerNext():135 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():119 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():109 > >>>> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext():91 > >>>> org.apache.drill.exec.record.AbstractRecordBatch.next():162 > >>>> org.apache.drill.exec.physical.impl.BaseRootExec.next():104 > >>>> org.apache.drill.exec.physical.impl.SingleSenderCreator$Sing > >>>> leSenderRootExec.innerNext():92 org.apache.drill.exec. > >> physical.impl.BaseRootExec.next():94 > >>>> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232 > >>>> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226 > >>>> java.security.AccessController.doPrivileged():-2 > >>>> javax.security.auth.Subject.doAs():422 org.apache.hadoop.security. > >> UserGroupInformation.doAs():1657 > >>>> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226 > >>>> org.apache.drill.common.SelfCleaningRunnable.run():38 > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker():1149 > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run():624 > >>>> java.lang.Thread.run():748 > >>>> > >>>> Dan Holmes | Revenue Analytics, Inc. > >>>> Direct: 770.859.1255 > >>>> www.revenueanalytics.com > >>>> > >>>> -----Original Message----- > >>>> From: Dan Holmes [mailto:[email protected]] > >>>> Sent: Thursday, July 27, 2017 9:11 AM > >>>> To: [email protected] > >>>> Subject: How much more information can error messages provide? > >>>> > >>>> I am querying a 6GB data set in a directory. There are 34 files in > the > >>>> directory. Here is the statement. > >>>> > >>>> CREATE TABLE dfs.root.`twm/sales_p` AS > >>>> SELECT to_date(columns[0], 'MM/dd/yy') transaction_date , > >> CAST(columns[1] > >>>> as INT) store_key , CAST(columns[2] AS BIGINT) item_code , columns[3] > >>>> position_name , CAST(CASE WHEN columns[4] = 'NULL' THEN 0 ELSE > >> columns[4] > >>>> END as INT) quantity_per_sku , CAST(columns[5] as INT) sales_quantity, > >>>> CAST(columns[6] as float) sales_dollars, columns[7] channel FROM > >>>> dfs.root.`sales` > >>>> > >>>> I get the following error. I was able to determine that columns[2] > >>>> needed to be a BIGINT based on an error and that columns[4] sometimes > >> had > >>>> the word "NULL" in it. In the case of NULL I had to run a statement > >> across > >>>> every column I was converting to INT (col1 = NULL or col2 = NULL ...) > to > >>>> figure out which one it was. Now I have a similar problem with the > >>>> TO_DATE() function. > >>>> > >>>> Can the error message include the filename and record number? Or at > >>>> least the value of what it found and which expression failed? Those > >> kinds > >>>> of clues would go a long way towards making this type of detective > work > >>>> significantly faster. The larger the dataset the faster it would be. > >>>> > >>>> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > >>>> IllegalArgumentException: Invalid format: " " Fragment 1:0 [Error Id: > >>>> 3557e893-e848-4503-bf6a-73110fbd6207 on ubuntu:31010] > >>>> > >>>> v1.10 > >>>> > >>>> Dan Holmes | Architect | Revenue Analytics, Inc. > >>>> 300 Galleria Parkway, Suite 1900 | Atlanta, Georgia 30339 > >>>> Direct: 770.859.1255 <(770)%20859-1255> Cell: 404.617.3444 > >>>> <(404)%20617-3444> > >>>> www.revenueanalytics.com<https://webmail.revenueanalytics.co > >>>> m/owa/redir.aspx?SURL=RqmyOJRm3r383jV2nPQLyg9BvjWZqMX4- > >>>> tL3BHj81WfaslMWau_SCGgAdAB0AHAAOgAvAC8AdwB3AHcALgByAGUAdgBlA > >>>> G4AdQBlAGEAbgBhAGwAeQB0AGkAYwBzAC4AYwBvAG0A&URL=http%3a%2f% > >>>> 2fwww.revenueanalytics.com> > >>>> LinkedIn<https://webmail.revenueanalytics.com/owa/redir. > >>>> aspx?SURL=SrcaeiXxVTCDhl49ibCO7CHhTsNynunc_8gSjHDaikXaslMWau_ > >> SCGgAdAB0AHA > >>>> AcwA6AC8ALwB3AHcAdwAuAGwAaQBuAGsAZQBkAGkAbgAuAGMAbwBtAC8AYwB > >>>> vAG0AcABhAG4AeQAvAHIAZQB2AGUAbgB1AGUALQBhAG4AYQBsAHkAdABpAGM > >>>> AcwAtAGkAbgBjAC0A&URL=https%3a%2f%2fwww.linkedin.com% > >>>> 2fcompany%2frevenue-analytics-inc-> | Twitter<https://webmail.revenu > >>>> eanalytics.com/owa/redir.aspx?SURL=cdePsMV8TCGx8O_Rugbj-maE9 > >>>> C9DVT373vSJwbUc23faslMWau_SCGgAdAB0AHAAcwA6AC8ALwB0AHcAaQB0A > >>>> HQAZQByAC4AYwBvAG0ALwBSAGUAdgBfAEEAbgBhAGwAeQB0AGkAYwBzAA..& > >>>> URL=https%3a%2f%2ftwitter.com%2fRev_Analytics> > >>>> > >>>> > >>> > >> > >
