subject:"\[jira\] \[Commented\] \(DRILL\-7385\) Convert PCAP Format Plugin to EVF"

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-15 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951832#comment-16951832
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-14 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951569#comment-16951569
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r334745856
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
 
 Review comment:
   Sounds good.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-14 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951568#comment-16951568
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r334746104
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  public static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-14 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951319#comment-16951319
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on issue #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#issuecomment-541896901
 
 
   @paul-rogers did Charles address all your code review comments? Is PR ready 
to be merged?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948684#comment-16948684
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on issue #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#issuecomment-540633178
 
 
   @cgivre thanks, yes, please squash the commits.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948682#comment-16948682
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on issue #1862: DRILL-7385: Convert PCAP Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#issuecomment-540632788
 
 
   Thanks @arina-ielchiieva for the review.   I added a comment for the TODO 
and created a JIRA: (https://issues.apache.org/jira/browse/DRILL-7400).  I will 
pre-emptively squash commits.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948676#comment-16948676
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on issue #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#issuecomment-540629893
 
 
   Overall LGTM, let's wait for Paul's approval as well.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948675#comment-16948675
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333571812
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948666#comment-16948666
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333561610
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948667#comment-16948667
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333561739
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948663#comment-16948663
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333561017
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948661#comment-16948661
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560624
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,72 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
+  public static final String PLUGIN_NAME = "pcap";
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-  @Override
-

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948660#comment-16948660
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560545
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapEVFReader.java
 ##
 @@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.categories.RowSetTests;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FileSystemPlugin;
+import org.apache.drill.test.BaseDirTestWatcher;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.time.LocalDateTime;
+import java.time.Month;
+
+
+@Category(RowSetTests.class)
+public class TestPcapEVFReader extends ClusterTest {
+
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
 
 Review comment:
   Removed...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948659#comment-16948659
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560465
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapEVFReader.java
 ##
 @@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.categories.RowSetTests;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FileSystemPlugin;
+import org.apache.drill.test.BaseDirTestWatcher;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.time.LocalDateTime;
+import java.time.Month;
+
+
+@Category(RowSetTests.class)
+public class TestPcapEVFReader extends ClusterTest {
+
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+
+  @BeforeClass
+  public static void setup() throws Exception {
+ClusterTest.startCluster(ClusterFixture.builder(dirTestWatcher));
+definePlugin();
+  }
+
+  private static void definePlugin() throws ExecutionSetupException {
+PcapFormatConfig sampleConfig = new PcapFormatConfig();
+
+// Define a temporary plugin for the "cp" storage plugin.
+Drillbit drillbit = cluster.drillbit();
+final StoragePluginRegistry pluginRegistry = 
drillbit.getContext().getStorage();
+final FileSystemPlugin plugin = (FileSystemPlugin) 
pluginRegistry.getPlugin("cp");
+final FileSystemConfig pluginConfig = (FileSystemConfig) 
plugin.getConfig();
+pluginConfig.getFormats().put("sample", sampleConfig);
+pluginRegistry.createOrUpdate("cp", pluginConfig, false);
+  }
+
+  @Test
+  public void testStarQuery() throws Exception {
+String sql = "SELECT * FROM cp.`store/pcap/synscan.pcap` LIMIT 1";
+
+testBuilder()
+  .sqlQuery(sql)
+  .unOrdered()
+  .baselineColumns("type", "packet_timestamp", "timestamp_micro", 
"network", "src_mac_address", "dst_mac_address", "dst_ip", "src_ip", 
"src_port", "dst_port", "packet_length",
+"tcp_session", "tcp_sequence", "tcp_ack", "tcp_flags", 
"tcp_parsed_flags", "tcp_flags_ns", "tcp_flags_cwr", "tcp_flags_ece", 
"tcp_flags_ece_ecn_capable", "tcp_flags_ece_congestion_experienced", 
"tcp_flags_urg", "tcp_flags_ack", "tcp_flags_psh", "tcp_flags_rst", 
"tcp_flags_syn", "tcp_flags_fin", "data", "is_corrupt")
+  .baselineValues("TCP", LocalDateTime.of(2010, Month.JULY, 4, 20, 24, 16, 
27400), 1278275056274870L, 1, "00:25:B3:BF:91:EE", "00:26:0B:31:07:33",
+"64.13.134.52", "172.16.0.8", 36050, 443, 58,
+317740574511239903L, -581795048, false, 2,"SYN", false, false, false, 
false, false, false, false, false, false, true, false,  "[]", false)
+  .build()
 
 Review comment:
   Done.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948658#comment-16948658
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560391
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +89,12 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948654#comment-16948654
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560053
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,25 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
 
-@JsonTypeName("pcap")
+import java.util.List;
+
+@JsonTypeName(PcapFormatPlugin.PLUGIN_NAME)
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = 
ImmutableList.of(PcapFormatPlugin.PLUGIN_NAME);
+
+  public List extensions;
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948655#comment-16948655
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560129
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatUtils.java
 ##
 @@ -74,6 +74,10 @@ public static int convertInt(final byte[] data, int offset) 
{
   }
 
   public static String parseBytesToASCII(byte[] data) {
+if(data == null) {
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948653#comment-16948653
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333559986
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948657#comment-16948657
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560300
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -26,15 +29,22 @@
 public class Schema {
 
   private final List columns = new ArrayList<>();
+  private MinorType typeMap[] = new MinorType[PcapTypes.values().length];
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948651#comment-16948651
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333559909
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948656#comment-16948656
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333560205
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatUtils.java
 ##
 @@ -74,6 +74,10 @@ public static int convertInt(final byte[] data, int offset) 
{
   }
 
   public static String parseBytesToASCII(byte[] data) {
+if(data == null) {
+  return "[]";
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948650#comment-16948650
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333559822
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948649#comment-16948649
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333559761
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948644#comment-16948644
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333556774
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948643#comment-16948643
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333556630
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948628#comment-16948628
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333543943
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948618#comment-16948618
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539731
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatUtils.java
 ##
 @@ -74,6 +74,10 @@ public static int convertInt(final byte[] data, int offset) 
{
   }
 
   public static String parseBytesToASCII(byte[] data) {
+if(data == null) {
 
 Review comment:
   ```suggestion
   if (data == null) {
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948626#comment-16948626
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333542789
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapEVFReader.java
 ##
 @@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.categories.RowSetTests;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FileSystemPlugin;
+import org.apache.drill.test.BaseDirTestWatcher;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.time.LocalDateTime;
+import java.time.Month;
+
+
+@Category(RowSetTests.class)
+public class TestPcapEVFReader extends ClusterTest {
+
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
+
+  @BeforeClass
+  public static void setup() throws Exception {
+ClusterTest.startCluster(ClusterFixture.builder(dirTestWatcher));
+definePlugin();
+  }
+
+  private static void definePlugin() throws ExecutionSetupException {
+PcapFormatConfig sampleConfig = new PcapFormatConfig();
+
+// Define a temporary plugin for the "cp" storage plugin.
+Drillbit drillbit = cluster.drillbit();
+final StoragePluginRegistry pluginRegistry = 
drillbit.getContext().getStorage();
+final FileSystemPlugin plugin = (FileSystemPlugin) 
pluginRegistry.getPlugin("cp");
+final FileSystemConfig pluginConfig = (FileSystemConfig) 
plugin.getConfig();
+pluginConfig.getFormats().put("sample", sampleConfig);
+pluginRegistry.createOrUpdate("cp", pluginConfig, false);
+  }
+
+  @Test
+  public void testStarQuery() throws Exception {
+String sql = "SELECT * FROM cp.`store/pcap/synscan.pcap` LIMIT 1";
+
+testBuilder()
+  .sqlQuery(sql)
+  .unOrdered()
+  .baselineColumns("type", "packet_timestamp", "timestamp_micro", 
"network", "src_mac_address", "dst_mac_address", "dst_ip", "src_ip", 
"src_port", "dst_port", "packet_length",
+"tcp_session", "tcp_sequence", "tcp_ack", "tcp_flags", 
"tcp_parsed_flags", "tcp_flags_ns", "tcp_flags_cwr", "tcp_flags_ece", 
"tcp_flags_ece_ecn_capable", "tcp_flags_ece_congestion_experienced", 
"tcp_flags_urg", "tcp_flags_ack", "tcp_flags_psh", "tcp_flags_rst", 
"tcp_flags_syn", "tcp_flags_fin", "data", "is_corrupt")
+  .baselineValues("TCP", LocalDateTime.of(2010, Month.JULY, 4, 20, 24, 16, 
27400), 1278275056274870L, 1, "00:25:B3:BF:91:EE", "00:26:0B:31:07:33",
+"64.13.134.52", "172.16.0.8", 36050, 443, 58,
+317740574511239903L, -581795048, false, 2,"SYN", false, false, false, 
false, false, false, false, false, false, true, false,  "[]", false)
+  .build()
 
 Review comment:
   Please use `go()` instead of `.build().run()`. Consider updating in other 
places...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948627#comment-16948627
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539992
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +89,12 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
 
 Review comment:
   ```suggestion
   for (ColumnDto column : columns) {
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948616#comment-16948616
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539623
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,25 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
 
-@JsonTypeName("pcap")
+import java.util.List;
+
+@JsonTypeName(PcapFormatPlugin.PLUGIN_NAME)
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = 
ImmutableList.of(PcapFormatPlugin.PLUGIN_NAME);
+
+  public List extensions;
 
 Review comment:
   Since you have added extensions, consider updating hadCode and equals 
methods.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948621#comment-16948621
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333538565
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948614#comment-16948614
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333538347
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948611#comment-16948611
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333537622
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
 
 Review comment:
   public
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948613#comment-16948613
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333538978
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948622#comment-16948622
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333544198
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948625#comment-16948625
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333543716
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948624#comment-16948624
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333543389
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,72 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
+  public static final String PLUGIN_NAME = "pcap";
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948619#comment-16948619
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539351
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948615#comment-16948615
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539852
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatUtils.java
 ##
 @@ -74,6 +74,10 @@ public static int convertInt(final byte[] data, int offset) 
{
   }
 
   public static String parseBytesToASCII(byte[] data) {
+if(data == null) {
+  return "[]";
 
 Review comment:
   Use constant instead.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948623#comment-16948623
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333540370
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapEVFReader.java
 ##
 @@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.categories.RowSetTests;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.dfs.FileSystemConfig;
+import org.apache.drill.exec.store.dfs.FileSystemPlugin;
+import org.apache.drill.test.BaseDirTestWatcher;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.time.LocalDateTime;
+import java.time.Month;
+
+
+@Category(RowSetTests.class)
+public class TestPcapEVFReader extends ClusterTest {
+
+  @ClassRule
+  public static final BaseDirTestWatcher dirTestWatcher = new 
BaseDirTestWatcher();
 
 Review comment:
   Why do you need to define `dirTestWatcher`? I though it already exists...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948620#comment-16948620
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539121
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948617#comment-16948617
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333539927
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -26,15 +29,22 @@
 public class Schema {
 
   private final List columns = new ArrayList<>();
+  private MinorType typeMap[] = new MinorType[PcapTypes.values().length];
 
 Review comment:
   ```suggestion
 private final MinorType typeMap[] = new 
MinorType[PcapTypes.values().length];
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-10 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948612#comment-16948612
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

arina-ielchiieva commented on pull request #1862: DRILL-7385: Convert PCAP 
Format Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r333538723
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private RowSetLoader rowWriter;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset;
+
+  private ScalarWriter typeWriter;
+
+  private ScalarWriter timestampWriter;
+
+  private ScalarWriter timestampMicroWriter;
+
+  private ScalarWriter networkWriter;
+
+  private ScalarWriter srcMacAddressWriter;
+
+  private ScalarWriter dstMacAddressWriter;
+
+  private ScalarWriter dstIPWriter;
+
+  private ScalarWriter srcIPWriter;
+
+  private ScalarWriter srcPortWriter;
+
+  private ScalarWriter dstPortWriter;
+
+  private ScalarWriter packetLengthWriter;
+
+  private ScalarWriter tcpSessionWriter;
+
+  private ScalarWriter tcpSequenceWriter;
+
+  private ScalarWriter tcpAckNumberWriter;
+
+  private ScalarWriter tcpFlagsWriter;
+
+  private ScalarWriter tcpParsedFlagsWriter;
+
+  private ScalarWriter tcpNsWriter;
+
+  private ScalarWriter tcpCwrWriter;
+
+  private ScalarWriter tcpEceWriter;
+
+  private ScalarWriter tcpFlagsEceEcnCapableWriter;
+
+  private ScalarWriter tcpFlagsCongestionWriter;
+
+  private ScalarWriter tcpUrgWriter;
+
+  private ScalarWriter tcpAckWriter;
+
+  private ScalarWriter tcpPshWriter;
+
+  private ScalarWriter tcpRstWriter;
+
+  private ScalarWriter tcpSynWriter;
+
+  private ScalarWriter tcpFinWriter;
+
+  private ScalarWriter dataWriter;
+
+  private ScalarWriter isCorruptWriter;
+
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+loader = negotiator.build();
+
+// Creates writers for all fields (Since schema is known)
+rowWriter = loader.writer();
+

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-07 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946384#comment-16946384
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r332305033
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-07 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945886#comment-16945886
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r332014039
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-07 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945863#comment-16945863
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r332001328
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapRecordReader.java
 ##
 @@ -45,8 +45,16 @@ public void testStarQuery() throws Exception {
   @Test
   public void testCorruptPCAPQuery() throws Exception {
 runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap`", 7000);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6408);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 592);
+  }
+
+  @Test
+  public void testTrueCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 16);
+  }
+
+  @Test
+  public void testNotCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6984);
 
 Review comment:
   I added new unit tests.  Also, I renamed the `timestamp` column to 
`packet_timestamp` so that the column name was not a Drill reserved word, which 
causes issues in queries.  
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1694#comment-1694
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331831324
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,71 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-  @Override
-  public boolean supportsStatistics() {
-return

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945556#comment-16945556
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331831337
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,27 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import java.util.List;
 
 @JsonTypeName("pcap")
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = ImmutableList.of("pcap");
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945552#comment-16945552
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828928
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapRecordReader.java
 ##
 @@ -45,8 +45,16 @@ public void testStarQuery() throws Exception {
   @Test
   public void testCorruptPCAPQuery() throws Exception {
 runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap`", 7000);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6408);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 592);
+  }
+
+  @Test
+  public void testTrueCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 16);
+  }
+
+  @Test
+  public void testNotCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6984);
 
 Review comment:
   Concur... I actually found a few undiscovered issues in doing the 
conversion.  I will add additional unit tests tomorrow. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945551#comment-16945551
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828873
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,27 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import java.util.List;
 
 @JsonTypeName("pcap")
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = ImmutableList.of("pcap");
+  public List extensions;
+
+  @JsonInclude(JsonInclude.Include.NON_DEFAULT)
+  public List getExtensions() {
+if (extensions == null) {
+  return DEFAULT_EXTS;
+}
+return extensions;
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945550#comment-16945550
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828852
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945548#comment-16945548
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828789
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945549#comment-16945549
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828808
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945546#comment-16945546
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828722
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.BIT);
+  } else if(column.getColumnType() == PcapTypes.INTEGER) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.INT);
+  } else if(column.getColumnType() == PcapTypes.STRING) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.VARCHAR);
+  } else if(column.getColumnType() == PcapTypes.TIMESTAMP) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.TIMESTAMP);
+  } else if(column.getColumnType() == PcapTypes.LONG) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.BIGINT);
+  }
+}
+
+TupleMetadata schema = builder.buildSchema();
+return schema;
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945545#comment-16945545
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828669
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.BIT);
+  } else if(column.getColumnType() == PcapTypes.INTEGER) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.INT);
+  } else if(column.getColumnType() == PcapTypes.STRING) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.VARCHAR);
+  } else if(column.getColumnType() == PcapTypes.TIMESTAMP) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.TIMESTAMP);
+  } else if(column.getColumnType() == PcapTypes.LONG) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.BIGINT);
 
 Review comment:
   I feel the need for speed... Fixed and went for the fastest solution. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945547#comment-16945547
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828747
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945540#comment-16945540
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828530
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
 
 Review comment:
   I'd agree.  However, the next step for this plugin is that I'm going to add 
packet dissectors.  Since there will likely be a performance hit, I'm going to 
also add a configuration option to disable them and then the reader config 
won't be useless.  So I'd like to leave it for now, even though at the moment, 
it serves no purpose. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945543#comment-16945543
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828579
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,71 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-  @Override
-  public boolean supportsStatistics() {
-return

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945541#comment-16945541
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828538
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945542#comment-16945542
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828558
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945544#comment-16945544
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331828597
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
 
 Review comment:
   Fixed
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945209#comment-16945209
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331765043
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945208#comment-16945208
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764842
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945214#comment-16945214
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331765111
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,71 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-  @Override
-  public boolean supportsStatistics() {
-

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945213#comment-16945213
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764819
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945206#comment-16945206
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764812
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945204#comment-16945204
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r328425281
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
 
 Review comment:
   No need for = 0
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945211#comment-16945211
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331767290
 
 

 ##
 File path: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/pcap/TestPcapRecordReader.java
 ##
 @@ -45,8 +45,16 @@ public void testStarQuery() throws Exception {
   @Test
   public void testCorruptPCAPQuery() throws Exception {
 runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap`", 7000);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6408);
-runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 592);
+  }
+
+  @Test
+  public void testTrueCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=true", 16);
+  }
+
+  @Test
+  public void testNotCorruptPCAPQuery() throws Exception {
+runSQLVerifyCount("select * from dfs.`store/pcap/testv1.pcap` WHERE 
is_corrupt=false", 6984);
 
 Review comment:
   Looks like the original test were pretty light. It leaves to the user to 
test per-column setup and type conversions.
   
   Would recommend adding tests that:
   
   1) Verify the schema: names, types
   2) Verifies the data (using the usual mechanism to read a few rows and 
validate the results.)
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945210#comment-16945210
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764799
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
 
 Review comment:
   Nit: The this. prefix is not needed.
 

This is an automated message from the

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945212#comment-16945212
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764958
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945216#comment-16945216
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r328425702
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
 
 Review comment:
   If this config holds only a single item, then it can be omitted. Just pass 
teh plugin into the batch reader instead of this class.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945217#comment-16945217
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331764935
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945215#comment-16945215
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331767130
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
 
 Review comment:
   Nit: space after if
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945205#comment-16945205
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331765065
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
+
+  public static class PcapReaderConfig {
+
+protected final PcapFormatPlugin plugin;
+public PcapReaderConfig(PcapFormatPlugin plugin) {
+  this.plugin = plugin;
+}
+  }
+
+  public PcapBatchReader(PcapReaderConfig readerConfig) {
+this.readerConfig = readerConfig;
+  }
+
+  @Override
+  public boolean open(FileSchemaNegotiator negotiator) {
+split = negotiator.split();
+openFile(negotiator);
+SchemaBuilder builder = new SchemaBuilder();
+this.pcapSchema = new Schema();
+TupleMetadata schema = pcapSchema.buildSchema(builder);
+negotiator.setTableSchema(schema, false);
+this.loader = negotiator.build();
+return true;
+  }
+
+  @Override
+  public boolean next() {
+RowSetLoader rowWriter = loader.writer();
+while (!rowWriter.isFull()) {
+  if (!parseNextPacket(rowWriter)) {
+return false;
+  }
+}
+return true;
+  }
+
+  @Override
+  public void close() {
+try {
+  fsStream.close();
+} catch (IOException e) {
+  throw UserException.
+dataReadError()
+.addContext("Error closing InputStream: " + e.getMessage())
+.build(logger);
+}
+fsStream = null;
+this.buffer = null;
+this.decoder = null;
+  }
+
+  private void openFile(FileSchemaNegotiator negotiator) {
+String filePath = null;
+try {
+  filePath = split.getPath().toString();
+  this.fsStream = negotiator.fileSystem().open(new Path(filePath));
+  this.decoder = new PacketDecoder(fsStream);
+  this.buffer = new byte[BUFFER_SIZE + decoder.getMaxLength()];
+  this.validBytes = fsStream.read(buffer);
+} catch (IOException io) {
+  throw UserException.dataReadError(io).addContext("File name:", 
filePath).build(logger);
+}
+  }
+
+  private boolean

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945219#comment-16945219
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331767252
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.BIT);
+  } else if(column.getColumnType() == PcapTypes.INTEGER) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.INT);
+  } else if(column.getColumnType() == PcapTypes.STRING) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.VARCHAR);
+  } else if(column.getColumnType() == PcapTypes.TIMESTAMP) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.TIMESTAMP);
+  } else if(column.getColumnType() == PcapTypes.LONG) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.BIGINT);
+  }
+}
+
+TupleMetadata schema = builder.buildSchema();
+return schema;
 
 Review comment:
   Nit: `return builder.buildSchema();`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945207#comment-16945207
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331767234
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/schema/Schema.java
 ##
 @@ -78,4 +81,23 @@ public ColumnDto getColumnByIndex(int i) {
   public int getNumberOfColumns() {
 return columns.size();
   }
+
+  public TupleMetadata buildSchema(SchemaBuilder builder) {
+for(ColumnDto column : columns) {
+  if(column.getColumnType() == PcapTypes.BOOLEAN) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.BIT);
+  } else if(column.getColumnType() == PcapTypes.INTEGER) {
+builder.addNullable(column.getColumnName(), TypeProtos.MinorType.INT);
+  } else if(column.getColumnType() == PcapTypes.STRING) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.VARCHAR);
+  } else if(column.getColumnType() == PcapTypes.TIMESTAMP) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.TIMESTAMP);
+  } else if(column.getColumnType() == PcapTypes.LONG) {
+builder.addNullable(column.getColumnName(), 
TypeProtos.MinorType.BIGINT);
 
 Review comment:
   Nit: would be simpler to do something like the following:
   
   ```
 for (ColumnTdo column : columns) {
   builder.addNullable(column.getColumnName(), convertType(column));
 }
   ...
   MinorType convertType(ColumnDto column) {
 switch (column.getColumnType()) {
   case PcapTypes.BOOLEAN:
 return MinorType.BIT;
  ...
   ```
   
   The above requires that we handle the case of a type that is not in the 
switch: maybe throw an exception or some such.
   
   Also, if the types are integers and configuous (which they are of PcapTypes 
is an enum), then you can create a mapping table:
   
   ```
 MinorType typeMap[] = new int[PcapTypes.getSize()];
 typeMap[PcapTypes.BOOLEAN.getOrdinal()] = MinorType.BIT;
 ...
   ```
   
   Then:
   
   ```
 for (ColumnTdo column : columns) {
   builder.addNullable(column.getColumnName(), 
typeMap[column.getColumType().getOrdinal());
   ```
   
   The mapping table is the fastest solution. But, this is not super critical 
in a once-per-file activity.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945222#comment-16945222
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331767121
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatPlugin.java
 ##
 @@ -17,112 +17,71 @@
  */
 package org.apache.drill.exec.store.pcap;
 
-import org.apache.drill.exec.planner.common.DrillStatsTable;
-import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
-import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileReaderFactory;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileScanBuilder;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.server.options.OptionManager;
+import org.apache.drill.exec.store.dfs.easy.EasySubScan;
 import org.apache.drill.common.exceptions.ExecutionSetupException;
-import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.common.logical.StoragePluginConfig;
-import org.apache.drill.exec.ops.FragmentContext;
-import org.apache.drill.exec.planner.logical.DrillTable;
 import org.apache.drill.exec.proto.UserBitShared;
 import org.apache.drill.exec.server.DrillbitContext;
-import org.apache.drill.exec.store.RecordReader;
-import org.apache.drill.exec.store.RecordWriter;
-import org.apache.drill.exec.store.SchemaConfig;
-import org.apache.drill.exec.store.dfs.BasicFormatMatcher;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.FileSystemPlugin;
-import org.apache.drill.exec.store.dfs.FormatMatcher;
-import org.apache.drill.exec.store.dfs.FormatSelection;
-import org.apache.drill.exec.store.dfs.MagicString;
 import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
-import org.apache.drill.exec.store.dfs.easy.EasyWriter;
-import org.apache.drill.exec.store.dfs.easy.FileWork;
 import org.apache.hadoop.conf.Configuration;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.regex.Pattern;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
+import org.apache.drill.exec.store.pcap.PcapBatchReader.PcapReaderConfig;
 
 public class PcapFormatPlugin extends EasyFormatPlugin {
 
-  private final PcapFormatMatcher matcher;
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf,
-  StoragePluginConfig storagePluginConfig) {
-this(name, context, fsConf, storagePluginConfig, new PcapFormatConfig());
-  }
-
-  public PcapFormatPlugin(String name, DrillbitContext context, Configuration 
fsConf, StoragePluginConfig config, PcapFormatConfig formatPluginConfig) {
-super(name, context, fsConf, config, formatPluginConfig, true, false, 
true, false, Lists.newArrayList("pcap"), "pcap");
-this.matcher = new PcapFormatMatcher(this);
-  }
-
-  @Override
-  public boolean supportsPushDown() {
-return true;
-  }
-
-  @Override
-  public RecordReader getRecordReader(FragmentContext context, DrillFileSystem 
dfs, FileWork fileWork, List columns, String userName) throws 
ExecutionSetupException {
-return new PcapRecordReader(fileWork.getPath(), dfs, columns);
-  }
-
-  @Override
-  public RecordWriter getRecordWriter(FragmentContext context, EasyWriter 
writer) throws IOException {
-throw new UnsupportedOperationException("unimplemented");
-  }
+  private static class PcapReaderFactory extends FileReaderFactory {
+private final PcapReaderConfig readerConfig;
 
-  @Override
-  public int getReaderOperatorType() {
-return UserBitShared.CoreOperatorType.PCAP_SUB_SCAN_VALUE;
-  }
+public PcapReaderFactory(PcapReaderConfig config) {
+  readerConfig = config;
+}
 
-  @Override
-  public int getWriterOperatorType() {
-throw new UnsupportedOperationException();
+@Override
+public ManagedReader newReader() {
+  return new PcapBatchReader(readerConfig);
+}
   }
-
-  @Override
-  public FormatMatcher getMatcher() {
-return this.matcher;
+  public PcapFormatPlugin(String name, DrillbitContext context,
+   Configuration fsConf, StoragePluginConfig 
storageConfig,
+   PcapFormatConfig formatConfig) {
+super(name, easyConfig(fsConf, formatConfig), context, storageConfig, 
formatConfig);
   }
 
-  @Override
-  public boolean supportsStatistics() {
-

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945221#comment-16945221
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331765083
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,27 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import java.util.List;
 
 @JsonTypeName("pcap")
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = ImmutableList.of("pcap");
 
 Review comment:
   Best if we define `"pcap"` to be a constant and use that constant here and 
above.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945218#comment-16945218
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r328425343
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapBatchReader.java
 ##
 @@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.pcap;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos;
+import 
org.apache.drill.exec.physical.impl.scan.file.FileScanFramework.FileSchemaNegotiator;
+import org.apache.drill.exec.physical.impl.scan.framework.ManagedReader;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.MetadataUtils;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.store.pcap.decoder.Packet;
+import org.apache.drill.exec.store.pcap.decoder.PacketDecoder;
+import org.apache.drill.exec.store.pcap.schema.Schema;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.mapred.FileSplit;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+
+import static 
org.apache.drill.exec.store.pcap.PcapFormatUtils.parseBytesToASCII;
+
+public class PcapBatchReader implements ManagedReader {
+
+  private FileSplit split;
+
+  private PcapReaderConfig readerConfig;
+
+  private PacketDecoder decoder;
+
+  private ResultSetLoader loader;
+
+  private FSDataInputStream fsStream;
+
+  private Schema pcapSchema;
+
+  private int validBytes;
+
+  private byte[] buffer;
+
+  private int offset = 0;
+
+  static final int BUFFER_SIZE = 500_000;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(PcapBatchReader.class);
 
 Review comment:
   Nit: statics usually go at the top of a class
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-10-05 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945220#comment-16945220
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

paul-rogers commented on pull request #1862: DRILL-7385: Convert PCAP Format 
Plugin to EVF
URL: https://github.com/apache/drill/pull/1862#discussion_r331765097
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapFormatConfig.java
 ##
 @@ -17,12 +17,27 @@
  */
 package org.apache.drill.exec.store.pcap;
 
+import com.fasterxml.jackson.annotation.JsonInclude;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.drill.common.logical.FormatPluginConfig;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableList;
+
+import java.util.List;
 
 @JsonTypeName("pcap")
 public class PcapFormatConfig implements FormatPluginConfig {
 
+  private static final List DEFAULT_EXTS = ImmutableList.of("pcap");
+  public List extensions;
+
+  @JsonInclude(JsonInclude.Include.NON_DEFAULT)
+  public List getExtensions() {
+if (extensions == null) {
+  return DEFAULT_EXTS;
+}
+return extensions;
 
 Review comment:
   Nit: `return extensions == null ? DEFAULT_EXTS : extensions`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (DRILL-7385) Convert PCAP Format Plugin to EVF

2019-09-24 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DRILL-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937281#comment-16937281
 ] 

ASF GitHub Bot commented on DRILL-7385:
---

cgivre commented on pull request #1862: DRILL-7385: Convert PCAP Format Plugin 
to EVF
URL: https://github.com/apache/drill/pull/1862
 
 
   The PCAP format plugin has the possibility of extracting considerably more 
data out of PCAP files. To facilitate this, this PR updates the plugin to use 
the new Enhanced Vector Framework. No changes in functionality should occur.
   
   I attempted to leave as much of the original code as possible. 
   
   @tdunning @paul-rogers 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Convert PCAP Format Plugin to EVF
> -
>
> Key: DRILL-7385
> URL: https://issues.apache.org/jira/browse/DRILL-7385
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.16.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> The PCAP format plugin has the possibility of extracting considerably more 
> data out of PCAP files.  To facilitate this, this PR updates the plugin to 
> use the new Enhanced Vector Framework.  No changes in functionality should 
> occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

81 matches

Mail list logo