subject:"\[GitHub\] drill pull request #654\: DRILL\-5032\: Drill query on hive parquet table faile..."

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-16 Thread Serhii-Harnyk

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92801855
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-16 Thread Serhii-Harnyk

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92801426
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-16 Thread Serhii-Harnyk

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92800877
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-15 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92305646
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-15 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92305881
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-15 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r92307044
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java
 ---
@@ -0,0 +1,483 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.hive;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.collect.Lists;
+
+@JsonTypeName("table")
+public class HiveTableWrapper {
+
+  @JsonIgnore
+  private HiveTable table;
+
+  @JsonProperty
+  public String tableName;
+  @JsonProperty
+  public String dbName;
+  @JsonProperty
+  public String owner;
+  @JsonProperty
+  public int createTime;
+  @JsonProperty
+  public int lastAccessTime;
+  @JsonProperty
+  public int retention;
+  @JsonProperty
+  public StorageDescriptorWrapper sd;
+  @JsonProperty
+  public List partitionKeys;
+  @JsonProperty
+  public Map parameters;
+  @JsonProperty
+  public String viewOriginalText;
+  @JsonProperty
+  public String viewExpandedText;
+  @JsonProperty
+  public String tableType;
+  @JsonProperty
+  public ColumnsCacheWrapper columnsCache;
+
+  @JsonIgnore
+  public final Map partitionNameTypeMap = new HashMap<>();
+
+  @JsonCreator
+  public HiveTableWrapper(@JsonProperty("tableName") String tableName, 
@JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner,
+  @JsonProperty("createTime") int createTime, 
@JsonProperty("lastAccessTime") int lastAccessTime,
+  @JsonProperty("retention") int retention, 
@JsonProperty("sd") StorageDescriptorWrapper sd,
+  @JsonProperty("partitionKeys") 
List partitionKeys, @JsonProperty("parameters") Map parameters,
+  @JsonProperty("viewOriginalText") String 
viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText,
+  @JsonProperty("tableType") String tableType, 
@JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache
+  ) {
+this.tableName = tableName;
+this.dbName = dbName;
+this.owner = owner;
+this.createTime = createTime;
+this.lastAccessTime = lastAccessTime;
+this.retention = retention;
+this.sd = sd;
+this.partitionKeys = partitionKeys;
+this.parameters = parameters;
+this.viewOriginalText = viewOriginalText;
+this.viewExpandedText = viewExpandedText;
+this.tableType = tableType;
+this.columnsCache = columnsCache;
+
+List partitionKeysUnwrapped = Lists.newArrayList();
+for (FieldSchemaWrapper w : partitionKeys) {
+  partitionKeysUnwrapped.add(w.getFieldSchema());
+  partitionNameTypeMap.put(w.name, w.type);
+}
+StorageDescriptor sdUnwrapped = sd.getSd();
+this.table = new HiveTable(tableName, dbName, owner, createTime, 
lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped,
+parameters, viewOriginalText, viewExpandedText, tableType, 
columnsCache.getColumnListsCache());
+  }
+
+  public HiveTableWrapper(HiveTable table) {
+if (table == null) {
+  return;
+}
+this.table = table;
+

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-07 Thread Serhii-Harnyk

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r91351247
  
--- Diff: 
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/schema/TestColumnListCache.java
 ---
@@ -0,0 +1,78 @@
+/*
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-06 Thread chunhui-shi

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r91153138
  
--- Diff: 
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/schema/TestColumnListCache.java
 ---
@@ -0,0 +1,78 @@
+/*
--- End diff --

Could you add a unit test to verify when there are two kind (or more) of 
columns in e.g. 10 partitions, the physical plan text has only two copies of 
columns?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-12-06 Thread chunhui-shi

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r91152465
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java
 ---
@@ -398,17 +399,13 @@ public static void addConfToJob(final JobConf job, 
final Properties properties)
* Wrapper around {@link MetaStoreUtils#getPartitionMetadata(Partition, 
Table)} which also adds parameters from table
* to properties returned by {@link 
MetaStoreUtils#getPartitionMetadata(Partition, Table)}.
*
-   * @param partition {@link Partition} instance
-   * @param table {@link Table} instance
+   * @param partition the source of partition level parameters
+   * @param table the source of table level parameters
* @return properties
*/
-  public static Properties getPartitionMetadata(final Partition partition, 
final Table table) {
+  public static Properties getPartitionMetadata(final HivePartition 
partition, final HiveTable table) {
 final Properties properties;
-// exactly the same column lists for partitions and table
-// stored only in table to reduce physical plan serialization
-if (partition.getSd().getCols() == null) {
-  partition.getSd().setCols(table.getSd().getCols());
-}
+restoreColumns(table, partition);
--- End diff --

could this restoreColumns fail the purpose of this fix? Since it is setting 
columns back to each partition, if getPartitionMetadata was called before the 
final physical plan generated, then these columns could be printed out to the 
plan again, could you check this possibility?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-11-24 Thread Serhii-Harnyk

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r89520154
  
--- Diff: 
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java
 ---
@@ -398,12 +398,18 @@ public static void addConfToJob(final JobConf job, 
final Properties properties)
* Wrapper around {@link MetaStoreUtils#getPartitionMetadata(Partition, 
Table)} which also adds parameters from table
* to properties returned by {@link 
MetaStoreUtils#getPartitionMetadata(Partition, Table)}.
*
-   * @param partition
-   * @param table
-   * @return
+   * @param partition {@link Partition} instance
+   * @param table {@link Table} instance
+   * @return properties
*/
   public static Properties getPartitionMetadata(final Partition partition, 
final Table table) {
-final Properties properties = 
MetaStoreUtils.getPartitionMetadata(partition, table);
+final Properties properties;
--- End diff --

@chunhui-shi let me clarify. Do you propose something like creating 
"partition groups" and storing columns in this "groups"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

2016-11-17 Thread Serhii-Harnyk

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/654

DRILL-5032: Drill query on hive parquet table failed with OutOfMemoryError



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-5032

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/654.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #654


commit 90482d7a1be99293fc3afdf2a297ee08e8831f66
Author: Serhii-Harnyk 
Date:   2016-10-27T19:20:27Z

DRILL-5032 Drill query on hive parquet table failed with OutOfMemoryError: 
Java heap space




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...

11 matches

Site Navigation

Mail list logo

Footer information