[GitHub] paul-rogers commented on issue #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

paul-rogers commented on issue #1455: DRILL-6724: Convert IndexOutOfBounds 
exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#issuecomment-418200233
 
 
   I have a basic question on this one. Is an Index Out Of Bounds Exception 
(IOBE) an expected behavior for this context? Or, does it indicate a bug in the 
code? Normally, code is expected to know the array size, and index only within 
that size, meaning that an IOBE is a program error.
   
   By presenting the error to the user in the nicely-formatted way introduced 
in this code, we seem to be implying that this is something the user needs to 
worry about. Yet, if this is a program error, we should say that this is a 
System Error and they need to contact Support for resolution.
   
   Note also that for unexpected program errors, there is a well-defined 
generic way we catch exceptions in the Fragment Executor. The Fragment Executor 
is the handler of last resort, will report an error, and will shut down the 
fragment.
   
   So, basic question: what is the source of the IOBE? If the IOBE is a program 
error, do we need the fancy handling if the Fragment Executor handles such 
errors? And, if the IOBE indicates a problem the user can resolve, should the 
error messages indicate a possible resolution? What might that problem be? 
Corrupted database? Incorrect SQL? Update done concurrently with a query? 
Something else?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Contrib Plugin Question

[Arina Yelchiyeva]

Personally, I did not have such problems with IntelliJ, did not try Eclipse, so 
can’t tell. Maybe somebody else can chime in.

Kind regards,
Arina 

> On Sep 3, 2018, at 10:23 PM, Paul Rogers  wrote:
> 
> I've been helping Charles with this. He's got a branch that works some times, 
> but not others.
> 
> * If I run his unit test from Eclipse, it works.
> * If I run his unit test from the command line with Maven, it works.
> * If he runs his unit test using the mechanism he is using, Drill can't find 
> his class.
> 
> The question is, has anyone who uses IntelliJ run into a similar issue? Do 
> you recall what you had to fix?
> 
> I'll continue to work with Charles to figure out what, exactly, he is doing 
> to run the test to see if we can zero in on the problem. But would be great 
> if someone said, "Yeah, I had that issue and I had to ..."
> 
> Thanks,
> - Paul
> 
> 
> 
>On Monday, September 3, 2018, 7:40:18 AM PDT, Arina Yelchiyeva 
>  wrote:  
> 
> Hi Charles.
> 
> Recently new udfs module was added under contrib, you can can take look at
> that PR for the example.
> 
> Regarding unit tests and data availability:
> 1. create TEST resources folder where you'll copy your data.
> 2. use dirTestWatcher to copy data to the root / tmp / custom directory.
> 3. query the data:
> 
> @Test
> public void t() {
>   dirTestWatcher.copyResourceToRoot(Paths.get("complex_1.parquet"));
>   queryBuilder().sql("select * from 
> dfs.`root`.`complex_1.parquet`").printCsv();
> }
> 
> 
> Also I would suggest you create module formats under contrib and place your
> format plugin module under formats: contrib / formats / format-plugin.
> 
>> On Mon, Sep 3, 2018 at 4:38 PM Charles Givre  wrote:
>> 
>> Hello all,
>> I’m working on a format-plugin for syslog (RFC-5424) data and I’m having
>> some strange issues.  I’d like to submit this contribution in the contrib/
>> folder, however I cannot seem to get Drill to recognize the module.  I’ve
>> built the module separately, and the code works, however, when I try to
>> build Drill, it does not recognize the module and I cannot query my data
>> (and the unit tests fail).  I’ve added the module to the contrib pom.xml
>> file and added the module to the assemble/bin.xml and still no luck.
>> Here’s the REALLY weird part.  I can run the unit tests in IntelliJ and it
>> works but it does not if I run the tests from the command line.
>> 
>> The code can be found here:
>> https://github.com/cgivre/drill/tree/format-syslog <
>> https://github.com/cgivre/drill/tree/format-syslog>
>> 
>> Does anyone have any suggestions?
>> 
>> —C
>> 
>> 

Re: Contrib Plugin Question

[Paul Rogers]

I've been helping Charles with this. He's got a branch that works some times, 
but not others.

* If I run his unit test from Eclipse, it works.
* If I run his unit test from the command line with Maven, it works.
* If he runs his unit test using the mechanism he is using, Drill can't find 
his class.

The question is, has anyone who uses IntelliJ run into a similar issue? Do you 
recall what you had to fix?

I'll continue to work with Charles to figure out what, exactly, he is doing to 
run the test to see if we can zero in on the problem. But would be great if 
someone said, "Yeah, I had that issue and I had to ..."

Thanks,
- Paul

 

On Monday, September 3, 2018, 7:40:18 AM PDT, Arina Yelchiyeva 
 wrote:  
 
 Hi Charles.

Recently new udfs module was added under contrib, you can can take look at
that PR for the example.

Regarding unit tests and data availability:
1. create TEST resources folder where you'll copy your data.
2. use dirTestWatcher to copy data to the root / tmp / custom directory.
3. query the data:

@Test
public void t() {
  dirTestWatcher.copyResourceToRoot(Paths.get("complex_1.parquet"));
  queryBuilder().sql("select * from dfs.`root`.`complex_1.parquet`").printCsv();
}


Also I would suggest you create module formats under contrib and place your
format plugin module under formats: contrib / formats / format-plugin.

On Mon, Sep 3, 2018 at 4:38 PM Charles Givre  wrote:

> Hello all,
> I’m working on a format-plugin for syslog (RFC-5424) data and I’m having
> some strange issues.  I’d like to submit this contribution in the contrib/
> folder, however I cannot seem to get Drill to recognize the module.  I’ve
> built the module separately, and the code works, however, when I try to
> build Drill, it does not recognize the module and I cannot query my data
> (and the unit tests fail).  I’ve added the module to the contrib pom.xml
> file and added the module to the assemble/bin.xml and still no luck.
> Here’s the REALLY weird part.  I can run the unit tests in IntelliJ and it
> works but it does not if I run the tests from the command line.
>
> The code can be found here:
> https://github.com/cgivre/drill/tree/format-syslog <
> https://github.com/cgivre/drill/tree/format-syslog>
>
> Does anyone have any suggestions?
>
> —C
>
>
>  

Re: [ANNOUNCE] New PMC member: Charles Givre

[Paul Rogers]

Congratulations Charles! I look forward to your continued strong voice as an 
expert Drill user in your new role.

- Paul

Sent from my iPhone

> On Sep 3, 2018, at 10:22 AM, Vitalii Diravka  wrote:
> 
> Congrats Charles!
> And thank you for your enthusiasm and work on Drill
> 
>> On Mon, Sep 3, 2018 at 4:22 PM Arina Ielchiieva  wrote:
>> 
>> I am pleased to announce that Drill PMC invited Charles Givre to the PMC
>> and he has accepted the invitation.
>> 
>> Congratulations Charles and welcome to PMC squad :)
>> 
>> - Arina
>> (on behalf of Drill PMC)
>> 

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214674475
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/bson/BsonRecordReader.java
 ##
 @@ -364,13 +367,14 @@ public void ensureAtLeastOneField(ComplexWriter writer) {
 }
   }
 
-  public UserException.Builder getExceptionWithContext(UserException.Builder 
exceptionBuilder, String field,
-  String msg, Object... args) {
-return null;
-  }
-
-  public UserException.Builder getExceptionWithContext(Throwable exception, 
String field, String msg, Object... args) {
-return null;
+  public UserException.Builder createExceptionWithContext(Throwable cause, 
String message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message);
+if (reader != null) {
+  builder.addContext("Name", reader.getCurrentName())
 
 Review comment:
   `addContext` from new line will look better


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: [ANNOUNCE] New PMC member: Charles Givre

[Vitalii Diravka]

Congrats Charles!
And thank you for your enthusiasm and work on Drill

On Mon, Sep 3, 2018 at 4:22 PM Arina Ielchiieva  wrote:

> I am pleased to announce that Drill PMC invited Charles Givre to the PMC
> and he has accepted the invitation.
>
> Congratulations Charles and welcome to PMC squad :)
>
> - Arina
> (on behalf of Drill PMC)
>

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548619
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/log/LogRecordReader.java
 ##
 @@ -761,4 +761,13 @@ public void close() {
   reader = null;
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+return UserException.dataReadError(cause)
+.message(message)
+.addContext("File", fileWork.getPath())
+.addContext("Line", rowIndex)
+.build(logger);
+  }
 }
 
 Review comment:
   nl


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548619
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/log/LogRecordReader.java
 ##
 @@ -761,4 +761,13 @@ public void close() {
   reader = null;
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+return UserException.dataReadError(cause)
+.message(message)
+.addContext("File", fileWork.getPath())
+.addContext("Line", rowIndex)
+.build(logger);
+  }
 }
 
 Review comment:
   nl


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548571
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java
 ##
 @@ -49,4 +50,15 @@
* @return The number of additional records added to the output.
*/
   int next();
+
+  /**
+   * Creates {@link UserException} with reader context. Should be analogous to
+   * {@link UserException#dataReadError(Throwable)} with provided context data.
+   * Returns {@code null} if context for a reader is unavailable.
 
 Review comment:
   Use `` or `` for the new line


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548608
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/sequencefile/SequenceFileRecordReader.java
 ##
 @@ -163,4 +163,23 @@ public void close() {
   logger.warn("Exception closing reader: {}", e);
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message)
+.addContext("File ", split.getPath().toString());
+long position = -1L;
+try {
+  if (reader != null) {
+position = reader.getPos();
+  }
+} catch (IOException e) {
+  // Ignore as this is to provide information only
+}
+if (position > 0) {
+  builder.addContext("Position ", position);
+}
+return builder.build(logger);
+  }
 }
 
 Review comment:
   Please add new line in the eof


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214706554
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/sequencefile/SequenceFileRecordReader.java
 ##
 @@ -163,4 +163,23 @@ public void close() {
   logger.warn("Exception closing reader: {}", e);
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message)
+.addContext("File ", split.getPath().toString());
+long position = -1L;
+try {
+  if (reader != null) {
+position = reader.getPos();
+  }
+} catch (IOException e) {
+  // Ignore as this is to provide information only
 
 Review comment:
   `logger.trace()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548665
 
 

 ##
 File path: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/HiveAbstractReader.java
 ##
 @@ -430,4 +431,23 @@ protected boolean hasNextValue(Object value) {
 }
   }
 
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message)
+.addContext("Database ", table.getDbName())
+.addContext("Table ", table.getTableName());
+long position = -1;
+try {
+  if (reader != null) {
+position = reader.getPos();
+  }
+} catch (IOException e) {
+  // Ignore as this is to provide information only
 
 Review comment:
   It would be good to log this exception message on TRACE level.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214697604
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/bson/BsonRecordReader.java
 ##
 @@ -364,13 +367,14 @@ public void ensureAtLeastOneField(ComplexWriter writer) {
 }
   }
 
-  public UserException.Builder getExceptionWithContext(UserException.Builder 
exceptionBuilder, String field,
-  String msg, Object... args) {
-return null;
-  }
-
-  public UserException.Builder getExceptionWithContext(Throwable exception, 
String field, String msg, Object... args) {
-return null;
+  public UserException.Builder createExceptionWithContext(Throwable cause, 
String message) {
 
 Review comment:
   It will be better to lead it to the same method name as 
`JsonProcessor.getExceptionWithContext()` for consistency.
   Because naturally in Mongo both `JsonReader` and `BsonBinaryreader` extend 
`AbstractBsonReader`
   
https://github.com/mongodb/mongo-java-driver/blob/master/bson/src/main/org/bson/json/JsonReader.java#L64
   Possibly it should be improved in Drill in future.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214706522
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java
 ##
 @@ -398,4 +399,21 @@ public void close() {
   }
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message)
+.addContext("File ", hadoop);
+long currentPosition = -1L;
+try {
+  currentPosition = reader.tell();
+} catch (IOException e) {
+  // Ignore as this is to provide information only
 
 Review comment:
   `logger.trace()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214681934
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/parquet/hadoop/ColumnChunkIncReadStore.java
 ##
 @@ -269,4 +270,10 @@ public PageReader getPageReader(ColumnDescriptor 
descriptor) {
   public long getRowCount() {
 return rowCount;
   }
+
+  public UserException.Builder getExceptionWithContext(Throwable cause, String 
message) {
+return UserException.dataReadError(cause)
 
 Review comment:
   Should it be in consistency with `DrillRuntimeException` in 117 line of this 
class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214687014
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java
 ##
 @@ -49,4 +50,15 @@
* @return The number of additional records added to the output.
*/
   int next();
+
+  /**
+   * Creates {@link UserException} with reader context. Should be analogous to
+   * {@link UserException#dataReadError(Throwable)} with provided context data.
+   * Returns {@code null} if context for a reader is unavailable.
+   *
+   * @param cause a {@code Throwable} to be wrapped into {@code UserException}
+   * @param message message to be passed to resulting {@code UserException}. 
Nullable
+   * @return exception with context data
 
 Review comment:
   just add `instance` word to clarify that it is not thrown, but returned.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214674475
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/bson/BsonRecordReader.java
 ##
 @@ -364,13 +367,14 @@ public void ensureAtLeastOneField(ComplexWriter writer) {
 }
   }
 
-  public UserException.Builder getExceptionWithContext(UserException.Builder 
exceptionBuilder, String field,
-  String msg, Object... args) {
-return null;
-  }
-
-  public UserException.Builder getExceptionWithContext(Throwable exception, 
String field, String msg, Object... args) {
-return null;
+  public UserException.Builder createExceptionWithContext(Throwable cause, 
String message) {
+UserException.Builder builder = UserException.dataReadError(cause)
+.message(message);
+if (reader != null) {
+  builder.addContext("Name", reader.getCurrentName())
 
 Review comment:
   `addContext` from new line


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214548521
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java
 ##
 @@ -20,14 +20,15 @@
 import java.util.Map;
 
 import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
 import org.apache.drill.exec.exception.OutOfMemoryException;
 import org.apache.drill.exec.ops.OperatorContext;
 import org.apache.drill.exec.physical.impl.OutputMutator;
 import org.apache.drill.exec.vector.ValueVector;
 
 public interface RecordReader extends AutoCloseable {
-  public static final long ALLOCATOR_INITIAL_RESERVATION = 1*1024*1024;
-  public static final long ALLOCATOR_MAX_RESERVATION = 20L*1000*1000*1000;
+  long ALLOCATOR_INITIAL_RESERVATION = 1*1024*1024;
+  long ALLOCATOR_MAX_RESERVATION = 20L*1000*1000*1000;
 
 Review comment:
   spaces


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214651648
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/sequencefile/SequenceFileRecordReader.java
 ##
 @@ -148,7 +148,7 @@ public int next() {
   return recordCount;
 } catch (IOException ioe) {
   close();
-  throw UserException.dataReadError(ioe).addContext("File Path", 
split.getPath().toString()).build(logger);
+  throw createDataReadException(ioe, null);
 
 Review comment:
   Looks like we will never get position value, so message will not be improved.
   Does it make sense to close reader after creating Data Read Exception?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdiravka commented on a change in pull request #1455: DRILL-6724: Convert IndexOutOfBounds exception to UserException with …

[GitBox]

vdiravka commented on a change in pull request #1455: DRILL-6724: Convert 
IndexOutOfBounds exception to UserException with …
URL: https://github.com/apache/drill/pull/1455#discussion_r214654837
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONRecordReader.java
 ##
 @@ -256,4 +256,13 @@ public void close() throws Exception {
   stream.close();
 }
   }
+
+  @Override
+  public UserException createDataReadException(Throwable cause, String 
message) {
 
 Review comment:
   should it be consistent with other Readers and used in `handleAndRaise` 
method or replace it?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Contrib Plugin Question

[Arina Yelchiyeva]

Hi Charles.

Recently new udfs module was added under contrib, you can can take look at
that PR for the example.

Regarding unit tests and data availability:
1. create TEST resources folder where you'll copy your data.
2. use dirTestWatcher to copy data to the root / tmp / custom directory.
3. query the data:

@Test
public void t() {
  dirTestWatcher.copyResourceToRoot(Paths.get("complex_1.parquet"));
  queryBuilder().sql("select * from dfs.`root`.`complex_1.parquet`").printCsv();
}


Also I would suggest you create module formats under contrib and place your
format plugin module under formats: contrib / formats / format-plugin.

On Mon, Sep 3, 2018 at 4:38 PM Charles Givre  wrote:

> Hello all,
> I’m working on a format-plugin for syslog (RFC-5424) data and I’m having
> some strange issues.  I’d like to submit this contribution in the contrib/
> folder, however I cannot seem to get Drill to recognize the module.  I’ve
> built the module separately, and the code works, however, when I try to
> build Drill, it does not recognize the module and I cannot query my data
> (and the unit tests fail).  I’ve added the module to the contrib pom.xml
> file and added the module to the assemble/bin.xml and still no luck.
> Here’s the REALLY weird part.  I can run the unit tests in IntelliJ and it
> works but it does not if I run the tests from the command line.
>
> The code can be found here:
> https://github.com/cgivre/drill/tree/format-syslog <
> https://github.com/cgivre/drill/tree/format-syslog>
>
> Does anyone have any suggestions?
>
> —C
>
>
>

Contrib Plugin Question

[Charles Givre]

Hello all, 
I’m working on a format-plugin for syslog (RFC-5424) data and I’m having some 
strange issues.  I’d like to submit this contribution in the contrib/ folder, 
however I cannot seem to get Drill to recognize the module.  I’ve built the 
module separately, and the code works, however, when I try to build Drill, it 
does not recognize the module and I cannot query my data (and the unit tests 
fail).  I’ve added the module to the contrib pom.xml file and added the module 
to the assemble/bin.xml and still no luck.  Here’s the REALLY weird part.  I 
can run the unit tests in IntelliJ and it works but it does not if I run the 
tests from the command line. 

The code can be found here: https://github.com/cgivre/drill/tree/format-syslog 


Does anyone have any suggestions?

—C 

[ANNOUNCE] New PMC member: Charles Givre

[Arina Ielchiieva]

I am pleased to announce that Drill PMC invited Charles Givre to the PMC
and he has accepted the invitation.

Congratulations Charles and welcome to PMC squad :)

- Arina
(on behalf of Drill PMC)

[GitHub] oleg-zinovev commented on a change in pull request #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM

[GitBox]

oleg-zinovev commented on a change in pull request #1446: DRILL-6349: Drill 
JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#discussion_r214607655
 
 

 ##
 File path: pom.xml
 ##
 @@ -734,6 +735,9 @@
   -Djava.net.preferIPv4Stack=true
   -Djava.awt.headless=true
   -XX:+CMSClassUnloadingEnabled -ea
+  -Djdk.attach.allowAttachSelf=true
+  -Duser.language=en
 
 Review comment:
   This tests fails on non-english locale. Main reason is a decimal and 
thousand separator char.
   https://drill.apache.org/docs/compiling-drill-from-source/ - I don't see any 
requirements about locale in documentaion. Maybe it must be an another Bug+PR ?
   
   TestDurationFormat.testCompactSecMillis:58->validateDurationFormat:43 
expected:<4[,]545s> but was:<4[.]545s>
   TestDurationFormat.testCompactTwoDigitMilliSec:48->validateDurationFormat:43 
expected:<0[,]045s> but was:<0[.]045s>
   TestFunctionsQuery.testToCharFunction:534 »  After matching 0 records, did 
not...
   TestSelectivity.testFilterSelectivityOptions » UserRemote PARSE ERROR: 
Encount...
   
TestVarDecimalFunctions.testCastDecimalDouble:680->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestVarDecimalFunctions.testCastDoubleDecimal:650->BaseTestQuery.testRunAndReturn:341
 » Rpc
   TestVarDecimalFunctions.testDecimalToChar:731 »  at position 0 column '`s1`' 
m...
   TestTopNSchemaChanges.testMissingColumn:192 »  
org.apache.drill.common.excepti...
   TestTopNSchemaChanges.testNumericTypes:82 »  
org.apache.drill.common.exception...
   TestTopNSchemaChanges.testUnionTypes:162 »  
org.apache.drill.common.exceptions...
   
TestMergeJoinWithSchemaChanges.testNumericStringTypes:192->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestMergeJoinWithSchemaChanges.testNumericTypes:114->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestMergeJoinWithSchemaChanges.testOneSideSchemaChanges:348->BaseTestQuery.testRunAndReturn:341
 » Rpc
   TestExternalSort.testNumericTypesLegacy:49->testNumericTypes:113 »  
org.apache...
   TestExternalSort.testNumericTypesManaged:44->testNumericTypes:113 »  
org.apach...
   TestImageRecordReader.testAviImage:101->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testBmpImage:56->createAndQuery:50 »  at position 0 
colu...
   TestImageRecordReader.testEpsImage:121->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testJpegImage:71->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testMovImage:111->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testPngImage:81->createAndQuery:50 »  at position 0 
colu...
   TestImageRecordReader.testPsdImage:86->createAndQuery:50 »  at position 0 
colu...
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] oleg-zinovev commented on a change in pull request #1446: DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM

[GitBox]

oleg-zinovev commented on a change in pull request #1446: DRILL-6349: Drill 
JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
URL: https://github.com/apache/drill/pull/1446#discussion_r214607655
 
 

 ##
 File path: pom.xml
 ##
 @@ -734,6 +735,9 @@
   -Djava.net.preferIPv4Stack=true
   -Djava.awt.headless=true
   -XX:+CMSClassUnloadingEnabled -ea
+  -Djdk.attach.allowAttachSelf=true
+  -Duser.language=en
 
 Review comment:
   This tests fails on non-english locale. Main reason is a decimal and 
thousand separator char.
   https://drill.apache.org/docs/compiling-drill-from-source/ - I don't see any 
requirements about locale in documentaion. Maybe it must be an another Bug+PR ?
   
   **TestDurationFormat.testCompactSecMillis:58->validateDurationFormat:43 
expected:<4[,]545s> but was:<4[.]545s>
   TestDurationFormat.testCompactTwoDigitMilliSec:48->validateDurationFormat:43 
expected:<0[,]045s> but was:<0[.]045s>
   TestFunctionsQuery.testToCharFunction:534 »  After matching 0 records, did 
not...
   TestSelectivity.testFilterSelectivityOptions » UserRemote PARSE ERROR: 
Encount...
   
TestVarDecimalFunctions.testCastDecimalDouble:680->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestVarDecimalFunctions.testCastDoubleDecimal:650->BaseTestQuery.testRunAndReturn:341
 » Rpc
   TestVarDecimalFunctions.testDecimalToChar:731 »  at position 0 column '`s1`' 
m...
   TestTopNSchemaChanges.testMissingColumn:192 »  
org.apache.drill.common.excepti...
   TestTopNSchemaChanges.testNumericTypes:82 »  
org.apache.drill.common.exception...
   TestTopNSchemaChanges.testUnionTypes:162 »  
org.apache.drill.common.exceptions...
   
TestMergeJoinWithSchemaChanges.testNumericStringTypes:192->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestMergeJoinWithSchemaChanges.testNumericTypes:114->BaseTestQuery.testRunAndReturn:341
 » Rpc
   
TestMergeJoinWithSchemaChanges.testOneSideSchemaChanges:348->BaseTestQuery.testRunAndReturn:341
 » Rpc
   TestExternalSort.testNumericTypesLegacy:49->testNumericTypes:113 »  
org.apache...
   TestExternalSort.testNumericTypesManaged:44->testNumericTypes:113 »  
org.apach...
   TestImageRecordReader.testAviImage:101->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testBmpImage:56->createAndQuery:50 »  at position 0 
colu...
   TestImageRecordReader.testEpsImage:121->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testJpegImage:71->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testMovImage:111->createAndQuery:50 »  at position 0 
col...
   TestImageRecordReader.testPngImage:81->createAndQuery:50 »  at position 0 
colu...
   TestImageRecordReader.testPsdImage:86->createAndQuery:50 »  at position 0 
colu...**
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services