[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154477971
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/BatchGroup.java
 ---
@@ -280,17 +278,14 @@ private void closeInputStream() throws IOException {
   logger.trace("Summary: Read {} bytes from {}", readLength, path);
 }
 
-public long closeOutputStream() throws IOException {
-  if (outputStream == null) {
-return 0;
+public void closeWriter() throws IOException {
+  if (writer == null) {
+return;
   }
-  long writeSize = spillSet.getPosition(outputStream);
-  spillSet.tallyWriteBytes(writeSize);
-  outputStream.close();
-  outputStream = null;
+  long writeSize = writer.close();
+  long timeNs = writer.timeNs();
--- End diff --

See comment above. Updating spill set here decouples the writer from the 
spill set. The `BatchGroup` already knows about the spill set.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154478390
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/cache/TestBatchSerialization.java
 ---
@@ -117,11 +117,12 @@ public void testNullableType(MinorType type) throws 
IOException {
   private void verifySerialize(SingleRowSet rowSet, SingleRowSet expected) 
throws IOException {
 
 File dir = OperatorFixture.getTempDir("serial");
-File outFile = new File(dir, "serialze.dat");
-try (OutputStream out = new BufferedOutputStream(new 
FileOutputStream(outFile))) {
-  VectorSerializer.writer(fixture.allocator(), out)
-.write(rowSet.container(), rowSet.getSv2());
-}
+VectorSerializer.writer(new SpillSet(dir.getAbsolutePath()),  
"serialize.dat")
--- End diff --

The `File dir` line should change to use Tim's new directory watcher 
feature. 


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154476694
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -62,27 +72,65 @@ public Writer write(VectorAccessible va) throws 
IOException {
 
 @SuppressWarnings("resource")
 public Writer write(VectorAccessible va, SelectionVector2 sv2) throws 
IOException {
+  checkNotNull(va);
   WritableBatch batch = WritableBatch.getBatchNoHVWrap(
   va.getRecordCount(), va, sv2 != null);
   return write(batch, sv2);
 }
 
 public Writer write(WritableBatch batch, SelectionVector2 sv2) throws 
IOException {
-  VectorAccessibleSerializable vas;
-  if (sv2 == null) {
-vas = new VectorAccessibleSerializable(batch, allocator);
-  } else {
-vas = new VectorAccessibleSerializable(batch, sv2, allocator);
-  }
-  if (retain) {
-vas.writeToStreamAndRetain(stream);
-  } else {
-vas.writeToStream(stream);
+  return write(batch, sv2, false);
+}
+
+public Writer write(WritableBatch batch, SelectionVector2 sv2, boolean 
retain) throws IOException {
--- End diff --

Retain is never actually used. That's why, in the earlier version, it was a 
flag that was not passed into the write method. Further, whether retain is not 
likely to be decided batch-by-batch, rather it is a general policy set at the 
writer level. So, if we don't like the `retain()` method, perhaps pass this 
option into the constructor rather than into each `write()` call.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154475785
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -62,27 +72,65 @@ public Writer write(VectorAccessible va) throws 
IOException {
 
 @SuppressWarnings("resource")
 public Writer write(VectorAccessible va, SelectionVector2 sv2) throws 
IOException {
+  checkNotNull(va);
   WritableBatch batch = WritableBatch.getBatchNoHVWrap(
   va.getRecordCount(), va, sv2 != null);
   return write(batch, sv2);
 }
 
 public Writer write(WritableBatch batch, SelectionVector2 sv2) throws 
IOException {
-  VectorAccessibleSerializable vas;
-  if (sv2 == null) {
-vas = new VectorAccessibleSerializable(batch, allocator);
-  } else {
-vas = new VectorAccessibleSerializable(batch, sv2, allocator);
-  }
-  if (retain) {
-vas.writeToStreamAndRetain(stream);
-  } else {
-vas.writeToStream(stream);
+  return write(batch, sv2, false);
+}
+
+public Writer write(WritableBatch batch, SelectionVector2 sv2, boolean 
retain) throws IOException {
+  checkNotNull(batch);
+  checkNotNull(channel);
+  final Timer.Context timerContext = 
metrics.timer(WRITER_TIMER).time();
+
+  final DrillBuf[] incomingBuffers = batch.getBuffers();
+  final UserBitShared.RecordBatchDef batchDef = batch.getDef();
+
+  try {
+/* Write the metadata to the file */
+batchDef.writeDelimitedTo(output);
+
+
+/* If we have a selection vector, dump it to file first */
--- End diff --

In the master branch, the `VectorSerializer` classes were slapped together 
as a simpler API on top of the existing `VectorAccessibleSerializable` classes. 
(That's why the code is so clunky.)

Here, we turn the `VectorSerializer` into the means to do the writing, 
which seems like a good improvement.

Does this mean we now have two implementations? The one here and the old 
one in `VectorAccessibleSerializable`? Do we need to clean up that split 
somehow? Else, should we leave comments to point people between the two copies 
so they know to make changes in both places?

Looking at the other changes, it seems we leave the old "unmanaged" 
external sort to use the old implementation.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154478770
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -62,27 +72,65 @@ public Writer write(VectorAccessible va) throws 
IOException {
 
 @SuppressWarnings("resource")
 public Writer write(VectorAccessible va, SelectionVector2 sv2) throws 
IOException {
+  checkNotNull(va);
   WritableBatch batch = WritableBatch.getBatchNoHVWrap(
   va.getRecordCount(), va, sv2 != null);
   return write(batch, sv2);
 }
 
 public Writer write(WritableBatch batch, SelectionVector2 sv2) throws 
IOException {
-  VectorAccessibleSerializable vas;
-  if (sv2 == null) {
-vas = new VectorAccessibleSerializable(batch, allocator);
-  } else {
-vas = new VectorAccessibleSerializable(batch, sv2, allocator);
-  }
-  if (retain) {
-vas.writeToStreamAndRetain(stream);
-  } else {
-vas.writeToStream(stream);
+  return write(batch, sv2, false);
+}
+
+public Writer write(WritableBatch batch, SelectionVector2 sv2, boolean 
retain) throws IOException {
+  checkNotNull(batch);
+  checkNotNull(channel);
+  final Timer.Context timerContext = 
metrics.timer(WRITER_TIMER).time();
+
+  final DrillBuf[] incomingBuffers = batch.getBuffers();
+  final UserBitShared.RecordBatchDef batchDef = batch.getDef();
+
+  try {
+/* Write the metadata to the file */
+batchDef.writeDelimitedTo(output);
+
+
+/* If we have a selection vector, dump it to file first */
+if (sv2 != null) {
+  final int dataLength = sv2.getCount() * 
SelectionVector2.RECORD_SIZE;
+  channel.write(sv2.getBuffer(false).nioBuffer(0, dataLength));
+}
+
+/* Dump the array of ByteBuf's associated with the value vectors */
+for (DrillBuf buf : incomingBuffers) {
+  /* dump the buffer into the OutputStream */
+  channel.write(buf.nioBuffer());
+}
+
+timeNs += timerContext.stop();
+  } catch (IOException e) {
+throw new RuntimeException(e);
+  } finally {
+if (!retain) {
--- End diff --

Frankly, the caller should decide what to do with the SV2 and the batch. 
Much simpler to just do:

```
writer.write(batch, sv2);
batch.clear();
sv2.clear();
```

Or, provide a `writeAndClear()` method if we think it is worth pulling 
those two lines into the write method.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154475350
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -62,27 +72,65 @@ public Writer write(VectorAccessible va) throws 
IOException {
 
 @SuppressWarnings("resource")
 public Writer write(VectorAccessible va, SelectionVector2 sv2) throws 
IOException {
+  checkNotNull(va);
   WritableBatch batch = WritableBatch.getBatchNoHVWrap(
   va.getRecordCount(), va, sv2 != null);
   return write(batch, sv2);
 }
 
 public Writer write(WritableBatch batch, SelectionVector2 sv2) throws 
IOException {
-  VectorAccessibleSerializable vas;
-  if (sv2 == null) {
-vas = new VectorAccessibleSerializable(batch, allocator);
-  } else {
-vas = new VectorAccessibleSerializable(batch, sv2, allocator);
-  }
-  if (retain) {
-vas.writeToStreamAndRetain(stream);
-  } else {
-vas.writeToStream(stream);
+  return write(batch, sv2, false);
+}
+
+public Writer write(WritableBatch batch, SelectionVector2 sv2, boolean 
retain) throws IOException {
+  checkNotNull(batch);
+  checkNotNull(channel);
+  final Timer.Context timerContext = 
metrics.timer(WRITER_TIMER).time();
+
+  final DrillBuf[] incomingBuffers = batch.getBuffers();
+  final UserBitShared.RecordBatchDef batchDef = batch.getDef();
+
+  try {
+/* Write the metadata to the file */
+batchDef.writeDelimitedTo(output);
+
+
+/* If we have a selection vector, dump it to file first */
+if (sv2 != null) {
+  final int dataLength = sv2.getCount() * 
SelectionVector2.RECORD_SIZE;
+  channel.write(sv2.getBuffer(false).nioBuffer(0, dataLength));
+}
+
+/* Dump the array of ByteBuf's associated with the value vectors */
+for (DrillBuf buf : incomingBuffers) {
+  /* dump the buffer into the OutputStream */
+  channel.write(buf.nioBuffer());
--- End diff --

While this technically works, it is a bit inefficient. Above, we write only 
the portion of the SV2 buffer that is actually used. But, here we grabbed 
buffers generically. Unless the write positions were set correctly for each 
buffer, we don't know the amount of the buffer that contains data. We may be 
writing the whole buffer, including unused internally-fragmented space.

If the write position is used to limit the write size, than the buffer 
write position should work just as well for the SV2, so we would not need the 
manual computation of used buffer size.

Perhaps the old code worked this way. But, I wonder if, since we are 
looking for greater performance, we should handle the issue here. (At present, 
Parquet produces batches with, say, 5% data and 95% unused. Do we want to spill 
all that space?)

Plus, since we spill the extra data, and we do power-of-two rounding on 
read, this probably explains why the sort must allow 2x the amount of memory 
for a deserialized spill batch than seems necessary from the sum of the data 
blocks.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154478289
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/BatchGroup.java
 ---
@@ -249,7 +247,7 @@ public void close() throws IOException {
 ex = e;
   }
   try {
-closeOutputStream();
+closeWriter();
--- End diff --

Just as a matter of record, this abstraction is rather awkward. It tries to 
be three things: 1) a writer for a spill file, 2) a reader for a spill file and 
3) the tombstone for the spill file.

It would be much better to have separate reader and writer classes that 
come and go, with this class just representing the spill file over its 
lifetime. There was, however, no easy way to make that change and preserve 
existing code. Since I'd already changed more code than I really wanted, I left 
well enough alone. (See the "unmanaged" version for the original code, which 
was even more murky.)

Now that you've modified the Writer to be the batch writer, I wonder if we 
should revisit the issue rather than preserving the old, muddy, semantics of 
this class.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154477573
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java
 ---
@@ -331,6 +337,43 @@ public long getReadBytes(InputStream inputStream) {
 }
   }
 
+  private static class WritableByteChannelImpl implements 
WritableByteChannel
+  {
+private static final int TRANSFER_SIZE = 32 * 1024;
+
+private OutputStream out;
+private final byte buffer[] = new byte[TRANSFER_SIZE];
--- End diff --

Unfortunately, this is one of the problems that the current design was 
intended to solve. As we can see in the hash agg code; multiple of these 
objects are active in memory at any one time. But, no more than one is ever 
writing. By putting the buffer here, we create one per (potentially thousands) 
of retained channels. But, by putting the buffer on the allocator, we hold one 
buffer per operator.

Being the guy who wrote the current allocator code, I realize it is clunky. 
But, it is clunky because we favored memory frugality over elegance. (If there 
is a cleaner way to have one buffer per thread, I'm all for it!)

Isn't the purpose of this change to avoid memory buffers? Seems, for the 
above reasons, we're moving backwards...


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154477089
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -62,27 +72,65 @@ public Writer write(VectorAccessible va) throws 
IOException {
 
 @SuppressWarnings("resource")
 public Writer write(VectorAccessible va, SelectionVector2 sv2) throws 
IOException {
+  checkNotNull(va);
   WritableBatch batch = WritableBatch.getBatchNoHVWrap(
   va.getRecordCount(), va, sv2 != null);
   return write(batch, sv2);
 }
 
 public Writer write(WritableBatch batch, SelectionVector2 sv2) throws 
IOException {
-  VectorAccessibleSerializable vas;
-  if (sv2 == null) {
-vas = new VectorAccessibleSerializable(batch, allocator);
-  } else {
-vas = new VectorAccessibleSerializable(batch, sv2, allocator);
-  }
-  if (retain) {
-vas.writeToStreamAndRetain(stream);
-  } else {
-vas.writeToStream(stream);
+  return write(batch, sv2, false);
+}
+
+public Writer write(WritableBatch batch, SelectionVector2 sv2, boolean 
retain) throws IOException {
+  checkNotNull(batch);
+  checkNotNull(channel);
+  final Timer.Context timerContext = 
metrics.timer(WRITER_TIMER).time();
+
+  final DrillBuf[] incomingBuffers = batch.getBuffers();
+  final UserBitShared.RecordBatchDef batchDef = batch.getDef();
+
+  try {
+/* Write the metadata to the file */
+batchDef.writeDelimitedTo(output);
+
+
+/* If we have a selection vector, dump it to file first */
+if (sv2 != null) {
+  final int dataLength = sv2.getCount() * 
SelectionVector2.RECORD_SIZE;
+  channel.write(sv2.getBuffer(false).nioBuffer(0, dataLength));
+}
+
+/* Dump the array of ByteBuf's associated with the value vectors */
+for (DrillBuf buf : incomingBuffers) {
+  /* dump the buffer into the OutputStream */
+  channel.write(buf.nioBuffer());
+}
+
+timeNs += timerContext.stop();
+  } catch (IOException e) {
+throw new RuntimeException(e);
+  } finally {
+if (!retain) {
+  batch.clear();
+  if (sv2 != null) {
+sv2.clear();
+  }
+}
   }
-  timeNs += vas.getTimeNs();
   return this;
 }
 
+public long close() throws IOException {
+  if (!channel.isOpen()) {
+return 0;
+  }
+  long writeSize = spillSet.getPosition(channel);
+  spillSet.tallyWriteBytes(writeSize);
--- End diff --

This seems awkward. Before, this class was a utility that wrapped an 
existing write mechanism. Here, this class basically becomes the writer itself 
-- a fancy file writer that manages its own channel. That seems like a good 
evolution.

But, it seems that updating spill set metrics is not needed here. Instead, 
someone will call this close method. That method can first update the spill set 
metrics based on the total write bytes for this stream. That is, this writer 
can do the writing, manage the channel, and accumulate its own write bytes. The 
caller decides what to do with that info (add it to the spill set's metrics.)


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154476440
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorSerializer.java 
---
@@ -40,20 +52,18 @@
*/
 
   public static class Writer {
+static final MetricRegistry metrics = DrillMetrics.getRegistry();
+static final String WRITER_TIMER = 
MetricRegistry.name(VectorAccessibleSerializable.class, "writerTime");
 
-private final OutputStream stream;
-private final BufferAllocator allocator;
-private boolean retain;
+private final SpillSet spillSet;
+private final WritableByteChannel channel;
+private final OutputStream output;
 private long timeNs;
 
-public Writer(BufferAllocator allocator, OutputStream stream) {
-  this.allocator = allocator;
-  this.stream = stream;
-}
-
-public Writer retain() {
-  retain = true;
-  return this;
+private Writer(SpillSet spillSet, String path) throws IOException {
--- End diff --

Before this class was independent of the `SpillSet`. Now, it can be used 
only with that class. Not sure if this is an improvement.

The only thing it seems we need from `SpillSet` is to update the write byte 
count. Perhaps define an interface here with just the needed methods. Then, let 
`SpillSet` implement that interface. That gets what you need without 
introducing extra dependencies.


---

[GitHub] drill pull request #1058: DRILL-6002: Avoid memory copy from direct buffer t...

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1058#discussion_r154477815
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java
 ---
@@ -419,6 +462,13 @@ public SpillSet(DrillConfig config, FragmentHandle 
handle, PhysicalOperator popC
 operName, handle.getMajorFragmentId(), popConfig.getOperatorId(), 
handle.getMinorFragmentId());
   }
 
+  @VisibleForTesting
+  public SpillSet(String baseDir) {
--- End diff --

We're trying to use the Hadoop `Path` class as our preferred way to pass 
file and directory names around.


---

[jira] [Created] (DRILL-6004) Direct buffer bounds checking should be disabled by default

[Vlad Rozov (JIRA)]

Vlad Rozov created DRILL-6004:
-

 Summary: Direct buffer bounds checking should be disabled by 
default
 Key: DRILL-6004
 URL: https://issues.apache.org/jira/browse/DRILL-6004
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Priority: Minor


Direct buffer bounds checking is enabled either when assertions are enabled 
(see DRILL-6001) or when {{drill.enable_unsafe_memory_access}} property is not 
set to true, so it is enabled in production as by default  
{{drill.enable_unsafe_memory_access}} is not set.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill issue #1057: DRILL-5993 Generic Copiers With Append Methods

[ilooner]

Github user ilooner commented on the issue:

https://github.com/apache/drill/pull/1057
  
@Ben-Zvi @paul-rogers As per our discussion I've added Paul's generic 
copiers and have provided a GenericCopier that works for RecordBatches that do 
not have selection vectors. I've enhanced the copier interface to include 
append methods.


---

Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Abhishek Girish]

I'm seeing a unit test failure tracked by DRILL-6003, when building from
[4]. Can someone take a look?

On Wed, Nov 29, 2017 at 6:50 AM, Arina Ielchiieva  wrote:

> Hi all,
>
> I'd like to propose the first release candidate (rc0) of Apache Drill,
> version 1.12.0.
>
> The release candidate covers a total of 167 resolved JIRAs [1]. Thanks to
> everyone who contributed to this release.
>
> The tarball artifacts are hosted at [2] and the maven artifacts are hosted
> at [3].
>
> This release candidate is based on commit
> 54d3d201882ef5bc2e0f754fd10edfead9947b60 located at [4].
>
> The vote ends at 3:00 PM UTC (7:00 AM PT), December 1, 2017.
>
> [ ] +1
> [ ] +0
> [ ] -1
>
> Here's my vote: +1
>
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12313820=12341087
> [2] http://home.apache.org/~arina/drill/releases/1.12.0/rc0/
> [3] https://repository.apache.org/content/repositories/
> orgapachedrill-1043/
> [4] https://github.com/arina-ielchiieva/drill/commits/drill-1.12.0
>
> Kind regards
> Arina
>

[jira] [Created] (DRILL-6003) Unit test testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport fails with FUNCTION ERROR: Failure reading Function class.

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-6003:
--

 Summary: Unit test 
testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport
 fails with FUNCTION ERROR: Failure reading Function class.
 Key: DRILL-6003
 URL: https://issues.apache.org/jira/browse/DRILL-6003
 Project: Apache Drill
  Issue Type: Bug
  Components: Tools, Build & Test
Affects Versions: 1.12.0
Reporter: Abhishek Girish


{code}
14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 B(1 
B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)): 
testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
org.apache.drill.exec.rpc.RpcException: 
org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: Failure 
reading Function class.

Function Class com.drill.udf.CustomLowerFunction
Fragment 0:0

[Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
at 
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60) 
~[drill-rpc-1.12.0.jar:1.12.0]
at 
org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
 ~[classes/:na]
at 
org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567) 
~[classes/:na]
at 
org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338) 
~[test-classes/:na]
at 
org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
 ~[test-classes/:na]
at 
org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
 ~[test-classes/:na]
at 
org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
 ~[test-classes/:na]
at 
org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147) 
~[test-classes/:na]
at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139) 
~[test-classes/:na]
at 
org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
 ~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[na:1.7.0_131]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: Failure 
reading Function class.

Function Class com.drill.udf.CustomLowerFunction
Fragment 0:0

[Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
at 
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
 ~[classes/:na]
at 
org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468) 
~[classes/:na]
at 
org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102) 
~[classes/:na]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274) 
~[drill-rpc-1.12.0.jar:1.12.0]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244) 
~[drill-rpc-1.12.0.jar:1.12.0]
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
 ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
 ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 

Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Paul Rogers]

Downloaded built gz file. Ran Drill. Exercised the queue-based memory 
allocation options and ran a query. Functionality and UI passed basic sanity 
tests. (UI tests are not yet automated, so this verification has to be done  by 
hand.) Krystal had done a full test earlier, which passed.

So, +1

- Paul

> On Nov 29, 2017, at 6:50 AM, Arina Ielchiieva  wrote:
> 
> Hi all,
> 
> I'd like to propose the first release candidate (rc0) of Apache Drill,
> version 1.12.0.
> 
> The release candidate covers a total of 167 resolved JIRAs [1]. Thanks to
> everyone who contributed to this release.
> 
> The tarball artifacts are hosted at [2] and the maven artifacts are hosted
> at [3].
> 
> This release candidate is based on commit
> 54d3d201882ef5bc2e0f754fd10edfead9947b60 located at [4].
> 
> The vote ends at 3:00 PM UTC (7:00 AM PT), December 1, 2017.
> 
> [ ] +1
> [ ] +0
> [ ] -1
> 
> Here's my vote: +1
> 
> 
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_ReleaseNote.jspa-3FprojectId-3D12313820-26version-3D12341087=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=Dz59a-Un_5n3KbQ2RYN0KA=mgZfNb5a1-f6BFp5aAo9f8Hmdx_RWwV8ZPRBTxYG8o8=JlFqOZ9UNzqySTtXJt7tYEpV45yA-5INWag82N8bFyQ=
> [2] 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Earina_drill_releases_1.12.0_rc0_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=Dz59a-Un_5n3KbQ2RYN0KA=mgZfNb5a1-f6BFp5aAo9f8Hmdx_RWwV8ZPRBTxYG8o8=v-nLG_AJl-qjcQgdAEII9RUWPWCwPKG-vLdg-8luq44=
> [3] 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_repositories_orgapachedrill-2D1043_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=Dz59a-Un_5n3KbQ2RYN0KA=mgZfNb5a1-f6BFp5aAo9f8Hmdx_RWwV8ZPRBTxYG8o8=KH_7NcwT8Pu0fJK4v9NXFsIz4uDg9ednPztD67S_Mu0=
> [4] 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_arina-2Dielchiieva_drill_commits_drill-2D1.12.0=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=Dz59a-Un_5n3KbQ2RYN0KA=mgZfNb5a1-f6BFp5aAo9f8Hmdx_RWwV8ZPRBTxYG8o8=jLcRIlRO06nUxENBpCndZQj9CUsd3N2g8CEr8WyURRw=
> 
> Kind regards
> Arina

Aw: RE: Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[at242]

 
No, this time it's 1.12 driver against 1.12 drillbit. The error was introduced 
in 1.11. If I build the driver from source, everthing is fine.

Gesendet: Freitag, 01. Dezember 2017 um 21:14 Uhr
Von: "Kunal Khatua" 
An: "dev@drill.apache.org" 
Betreff: RE: Re: [VOTE] Release Apache Drill 1.12.0 - rc0
Are you trying to connect to pre-1.11.0 server? The JIRA mentions you swapped 
out Drill 1.10 JDBC clients with 1.11.0 and hit an issue.

-Original Message-
From: korlawu...@die-optimisten.net [mailto:korlawu...@die-optimisten.net]
Sent: Friday, December 01, 2017 12:11 PM
To: dev@drill.apache.org
Subject: Aw: Re: [VOTE] Release Apache Drill 1.12.0 - rc0


-1
 
Because the prebuild drill-jdbc-driver is not usable:

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D5702=DwIFaQ=cskdkSMqhcnjZxdQVpwTXg=-cT6otg6lpT_XkmYy7yg3A=hceP_7Wz7DFolt7fwNlahzO1Vg5eib21n1yawyZz0ns=p4FzUFV12qeUAq7k7fbrZYuZD6tjieiBZlmQZX6Wanw=

RE: Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Kunal Khatua]

Are you trying to connect to pre-1.11.0 server? The JIRA mentions you swapped 
out Drill 1.10 JDBC clients with 1.11.0 and hit an issue. 

-Original Message-
From: korlawu...@die-optimisten.net [mailto:korlawu...@die-optimisten.net] 
Sent: Friday, December 01, 2017 12:11 PM
To: dev@drill.apache.org
Subject: Aw: Re: [VOTE] Release Apache Drill 1.12.0 - rc0


-1
 
Because the prebuild drill-jdbc-driver is not usable:

https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D5702=DwIFaQ=cskdkSMqhcnjZxdQVpwTXg=-cT6otg6lpT_XkmYy7yg3A=hceP_7Wz7DFolt7fwNlahzO1Vg5eib21n1yawyZz0ns=p4FzUFV12qeUAq7k7fbrZYuZD6tjieiBZlmQZX6Wanw=

Aw: Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[korlawulki]

-1
 
Because the prebuild drill-jdbc-driver is not usable:

https://issues.apache.org/jira/browse/DRILL-5702

Re: [VOTE] Release Apache Drill 1.12.0 - rc0

[Arina Yelchiyeva]

Sorry, incorrectly estimated end date, it should be at 3:00 PM UTC (7:00 AM
PST), December 2, 2017.
So far we don't have enough votes to accept the release candidate. Please
vote!

Kind regards
Arina


On Thu, Nov 30, 2017 at 9:36 PM, Kunal Khatua  wrote:

> Performed sanity testing:
> Set up Drill on a physical node and ran a couple of queries consisting of
> Joins, aggregations, etc against the TPCH dataset.
>
> +1 (non-binding)
>
>
> -Original Message-
> From: Aman Sinha [mailto:amansi...@apache.org]
> Sent: Thursday, November 30, 2017 8:52 AM
> To: dev@drill.apache.org
> Subject: Re: [VOTE] Release Apache Drill 1.12.0 - rc0
>
> - Downloaded the source tarball from [2] on my Linux VM
> - Built and ran all unit tests - took about 36 minutes.  Monitored the
> memory usage on the machine periodically, looked good.
> - Started Drill in embedded mode and manually ran a bunch of sanity
> queries with joins, aggregations, order-by and CTAS statements, against a
> TPCH dataset.
> - Examined the run-time query profiles of several queries with parallelism.
> - Checked the maven artifacts on [3].
>
> LGTM  +1 (binding)
>
> Thanks Arina for guiding the release and thanks to all contributors who
> worked on the release !
>
>
> On Wed, Nov 29, 2017 at 6:50 AM, Arina Ielchiieva 
> wrote:
>
> > Hi all,
> >
> > I'd like to propose the first release candidate (rc0) of Apache Drill,
> > version 1.12.0.
> >
> > The release candidate covers a total of 167 resolved JIRAs [1]. Thanks
> > to everyone who contributed to this release.
> >
> > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted at [3].
> >
> > This release candidate is based on commit
> > 54d3d201882ef5bc2e0f754fd10edfead9947b60 located at [4].
> >
> > The vote ends at 3:00 PM UTC (7:00 AM PT), December 1, 2017.
> >
> > [ ] +1
> > [ ] +0
> > [ ] -1
> >
> > Here's my vote: +1
> >
> >
> > [1]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org
> > _jira_secure_ReleaseNote.jspa-3F=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=-
> > cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU=
> > G7nd0QG8searR-Ynr_L0uwLMLVPPrZQM7-kLDX6CzLg=
> > projectId=12313820=12341087
> > [2]
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7
> > Earina_drill_releases_1.12.0_rc0_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=
> > -cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU
> > =Wip_bHewQWUXZfGZbZcWpv2BtSkeSHIuyxaDwBOnwXA=
> > [3]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache
> > .org_content_repositories_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=-cT6otg
> > 6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL-NRZVU=A2sU7W
> > RWS4GRHVQeFf3oKlT1iWB1bxkXtmrZcQHVqT8=
> > orgapachedrill-1043/
> > [4]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_arina-
> > 2Dielchiieva_drill_commits_drill-2D1.12.0=DwIBaQ=cskdkSMqhcnjZxdQV
> > pwTXg=-cT6otg6lpT_XkmYy7yg3A=NVHw0r_esxwKsdEOgV3LQnZ3vYijKSHbE_gjL
> > -NRZVU=W9VClczvLQyBXfLsATbXDCX8QxIRddSH8ap3kyU5YVY=
> >
> > Kind regards
> > Arina
> >
>