[
https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246442#comment-16246442
]
ASF GitHub Bot commented on DRILL-5783:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/984#discussion_r150072992
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/file/JsonFileBuilder.java
---
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.test.rowSet.file;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.vector.accessor.ColumnAccessor;
+import org.apache.drill.exec.vector.accessor.ColumnReader;
+import org.apache.drill.test.rowSet.RowSet;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+public class JsonFileBuilder
+{
+ public static final String DEFAULT_DOUBLE_FORMATTER = "%f";
+ public static final String DEFAULT_INTEGER_FORMATTER = "%d";
+ public static final String DEFAULT_LONG_FORMATTER = "%d";
+ public static final String DEFAULT_STRING_FORMATTER = "\"%s\"";
+ public static final String DEFAULT_DECIMAL_FORMATTER = "%s";
+ public static final String DEFAULT_PERIOD_FORMATTER = "%s";
+
+ public static final Map<String, String> DEFAULT_FORMATTERS = new
ImmutableMap.Builder()
+ .put(ColumnAccessor.ValueType.DOUBLE, DEFAULT_DOUBLE_FORMATTER)
+ .put(ColumnAccessor.ValueType.INTEGER, DEFAULT_INTEGER_FORMATTER)
+ .put(ColumnAccessor.ValueType.LONG, DEFAULT_LONG_FORMATTER)
+ .put(ColumnAccessor.ValueType.STRING, DEFAULT_STRING_FORMATTER)
+ .put(ColumnAccessor.ValueType.DECIMAL, DEFAULT_DECIMAL_FORMATTER)
+ .put(ColumnAccessor.ValueType.PERIOD, DEFAULT_PERIOD_FORMATTER)
+ .build();
+
+ private final RowSet rowSet;
+ private final Map<String, String> customFormatters = Maps.newHashMap();
+
+ public JsonFileBuilder(RowSet rowSet) {
+ this.rowSet = Preconditions.checkNotNull(rowSet);
+ Preconditions.checkArgument(rowSet.rowCount() > 0, "The given rowset
is empty.");
+ }
+
+ public JsonFileBuilder setCustomFormatter(final String columnName, final
String columnFormatter) {
+ Preconditions.checkNotNull(columnName);
+ Preconditions.checkNotNull(columnFormatter);
+
+ Iterator<MaterializedField> fields = rowSet
+ .schema()
+ .batch()
+ .iterator();
+
+ boolean hasColumn = false;
+
+ while (!hasColumn && fields.hasNext()) {
+ hasColumn = fields.next()
+ .getName()
+ .equals(columnName);
+ }
+
+ final String message = String.format("(%s) is not a valid column",
columnName);
+ Preconditions.checkArgument(hasColumn, message);
+
+ customFormatters.put(columnName, columnFormatter);
+
+ return this;
+ }
+
+ public void build(File tableFile) throws IOException {
--- End diff --
Great! This does not yet handle nested tuples or arrays; in part because
the row set work for that is still sitting in PR #914. You can update this to
be aware of maps and map arrays once that PR is committed.
> Make code generation in the TopN operator more modular and test it
> ------------------------------------------------------------------
>
> Key: DRILL-5783
> URL: https://issues.apache.org/jira/browse/DRILL-5783
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
>
> The work for this PR has had several other PRs batched together with it. The
> full description of work is the following:
> DRILL-5783
> * A unit test is created for the priority queue in the TopN operator
> * The code generation classes passed around a completely unused function
> registry reference in some places so I removed it.
> * The priority queue had unused parameters for some of its methods so I
> removed them.
> DRILL-5841
> * There were many many ways in which temporary folders were created in unit
> tests. I have unified the way these folders are created with the
> DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests
> have been updated to use these. The test watchers create temp directories in
> ./target//. So all the files generated and used in the context of a test can
> easily be found in the same consistent location.
> * This change should fix the sporadic hashagg test failures, as well as
> failures caused by stray files in /tmp
> DRILL-5894
> * dfs_test is used as a storage plugin throughout the unit tests. This is
> highly confusing and we can just use dfs instead.
> *Misc*
> * General code cleanup.
> * There are many places where String.format is used unnecessarily. The test
> builder methods already use String.format for you when you pass them args. I
> cleaned some of these up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
