[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1057 ---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r163028998
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/GenericSV4Copier.java
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.svremover;
+
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.vector.SchemaChangeCallBack;
+import org.apache.drill.exec.vector.ValueVector;
+
+public class GenericSV4Copier extends AbstractSV4Copier {
+
+ @Override
+ public void setup(RecordBatch incoming, VectorContainer outgoing) throws
SchemaChangeException {
+super.setup(incoming, outgoing);
--- End diff --
The code is similar but not identical. For **GenericSV4Copier** **vvIn** is
a 2D array of type **ValueVector[][]**. For **GenericSV2Copier** **vvIn** is a
1D array of type **ValueVector[]**. Since the types are different I can't move
the code into the **AbstractCopier**. But I will move the setup code in
**GenericSV2Copier** to **AbstractSV2Copier** since that code will likely be
the same for different implementations. I'll also do the same thing with the
**GenericSV4Copier** and **AbstractSV4Copier** classes.
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r163025549
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java ---
@@ -60,6 +60,10 @@ public SelectionVectorMode getSelectionVectorMode() {
return selectionVectorMode;
}
+ public BatchSchema toMode(SelectionVectorMode mode) {
--- End diff --
removed
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r162771977
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java ---
@@ -60,6 +60,10 @@ public SelectionVectorMode getSelectionVectorMode() {
return selectionVectorMode;
}
+ public BatchSchema toMode(SelectionVectorMode mode) {
--- End diff --
Unused code ?
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r162771857
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/GenericSV4Copier.java
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.svremover;
+
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.vector.SchemaChangeCallBack;
+import org.apache.drill.exec.vector.ValueVector;
+
+public class GenericSV4Copier extends AbstractSV4Copier {
+
+ @Override
+ public void setup(RecordBatch incoming, VectorContainer outgoing) throws
SchemaChangeException {
+super.setup(incoming, outgoing);
--- End diff --
Can the setup() code in both GenericSV2Copier and GenericSV4Copier be moved
up into the setup() method of AbstractCopier ? Both codes are identical
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r162196004
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/AbstractCopier.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.svremover;
+
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+
+public abstract class AbstractCopier implements Copier {
+ protected ValueVector[] vvOut;
+ protected VectorContainer outgoing;
+
+ @Override
+ public void setup(RecordBatch incoming, VectorContainer outgoing) throws
SchemaChangeException {
--- End diff --
It is not used by the AbstractCopier itself but it is used by the
AbstractSV2Copier and AbstractSV4Copier which override the method.
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r162193983
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/AbstractCopier.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.svremover;
+
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+
+public abstract class AbstractCopier implements Copier {
+ protected ValueVector[] vvOut;
+ protected VectorContainer outgoing;
+
+ @Override
+ public void setup(RecordBatch incoming, VectorContainer outgoing) throws
SchemaChangeException {
--- End diff --
The "incoming" parameter is not used anywhere ...
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r162175059
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/AbstractSV4Copier.java
---
@@ -17,32 +17,30 @@
*/
package org.apache.drill.exec.physical.impl.svremover;
-import javax.inject.Named;
-
import org.apache.drill.common.types.TypeProtos.MajorType;
import org.apache.drill.common.types.Types;
import org.apache.drill.exec.exception.SchemaChangeException;
-import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
import org.apache.drill.exec.record.VectorWrapper;
import org.apache.drill.exec.record.selection.SelectionVector4;
import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
-public abstract class CopierTemplate4 implements Copier{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(CopierTemplate4.class);
+public abstract class AbstractSV4Copier implements Copier {
--- End diff --
Done
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r161950855
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/AbstractSV4Copier.java
---
@@ -17,32 +17,30 @@
*/
package org.apache.drill.exec.physical.impl.svremover;
-import javax.inject.Named;
-
import org.apache.drill.common.types.TypeProtos.MajorType;
import org.apache.drill.common.types.Types;
import org.apache.drill.exec.exception.SchemaChangeException;
-import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.VectorContainer;
import org.apache.drill.exec.record.VectorWrapper;
import org.apache.drill.exec.record.selection.SelectionVector4;
import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
-public abstract class CopierTemplate4 implements Copier{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(CopierTemplate4.class);
+public abstract class AbstractSV4Copier implements Copier {
--- End diff --
The code for AbstractSV4Copier is nearly identical to the code of
AbstractSV2Copier; I wonder if we could have combined the two somehow (like a
super class ??)
---
[GitHub] drill pull request #1057: DRILL-5993 Add Generic Copiers With Append Methods
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1057#discussion_r161913565
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/svremover/GenericCopierTest.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.svremover;
+
+import org.apache.drill.exec.memory.RootAllocator;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.test.rowSet.RowSet;
+import org.apache.drill.test.rowSet.RowSetBuilder;
+
+public class GenericCopierTest extends AbstractGenericCopierTest {
--- End diff --
The generated code uses the copyFromSafe methods in code generated copiers.
Your copyEntry method which is used by the GenericCopiers also uses the
copyFromSafe method. So we should inherit all the same supported types. I think
if all the unit tests, functional tests, and QA tests pass that will be a
sufficient enough vote of confidence. Ideally we would have unit tests for all
the operators (not just the copiers) which use the complex data types, but
there is significant amount of work left to be able to effectively unit tests
with complex data types and adding all the machinery to do so is out of the
scope of this PR.
---
