[
https://issues.apache.org/jira/browse/APEXMALHAR-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219817#comment-15219817
]
ASF GitHub Bot commented on APEXMALHAR-2014:
--------------------------------------------
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/219#discussion_r58045703
--- Diff:
contrib/src/main/java/com/datatorrent/contrib/parquet/ParquetFilePOJOReader.java
---
@@ -0,0 +1,318 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package com.datatorrent.contrib.parquet;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.StringTokenizer;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.ClassUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultOutputPort;
+import com.datatorrent.api.annotation.OutputPortFieldAnnotation;
+import com.datatorrent.lib.util.FieldInfo;
+import com.datatorrent.lib.util.FieldInfo.SupportType;
+import com.datatorrent.lib.util.PojoUtils;
+import com.datatorrent.lib.util.PojoUtils.Setter;
+
+import parquet.example.data.Group;
+import parquet.io.InvalidRecordException;
+import parquet.io.ParquetEncodingException;
+import parquet.schema.PrimitiveType.PrimitiveTypeName;
+
+/**
+ * <p>
+ * ParquetFilePOJOReader
+ * </p>
+ * ParquetFilePOJOReader operator is a concrete implementation of
+ * AbstractParquetFileReader to read Parquet files and emit records as
POJOs.The
+ * POJO class name & field mapping should be provided by the user. If this
+ * mapping is not provided then reflection is used to determine this
mapping. As
+ * of now only primitive types ( INT32, INT64, BOOLEAN, FLOAT, DOUBLE,
BINARY )
+ * are supported.
+ *
+ * Parquet primitive types maps to POJO datatypes as follows : INT32 -> int
+ * INT64 -> long BOOLEAN -> boolean FLOAT -> float DOUBLE -> double BINARY
->
+ * String
+ *
+ * @displayName ParquetFilePOJOReader
+ * @tags parquet,input adapter
+ * @since 3.3.0
+ */
+public class ParquetFilePOJOReader extends
AbstractParquetFileReader<Object>
+{
+
+ /**
+ * POJO class
+ */
+ protected transient Class<?> pojoClass;
+ /**
+ * Map containing setters for fields in POJO
+ */
+ protected transient Map<String, Setter> pojoSetters;
+ /**
+ * String representing Parquet TO POJO field mapping. If not provided,
then
+ * reflection is used to determine the mapping. Format :
+ * PARQUET_FIELD_NAME:POJO_FIELD_NAME:TYPE
+ * E.g.event_id:event_id_v2:INTEGER,org_id:org_id_v2:STRING,long_id:
+ * long_id_v2:
+ *
LONG,css_file_loaded:css_file_loaded_v2:BOOLEAN,float_val:float_val_v2:
+ * FLOAT,double_val:double_val_v2:DOUBLE
+ */
+ protected transient String groupToPOJOFieldsMapping = null;
+ protected transient List<FieldInfo> fieldInfos;
+ protected transient List<ActiveFieldInfo> columnFieldSetters = null;
--- End diff --
I don't see columnFieldSetters been any dfferent from fieldInfos variable.
Can they be merged to single variable?
> ParquetReader operator
> ----------------------
>
> Key: APEXMALHAR-2014
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2014
> Project: Apache Apex Malhar
> Issue Type: New Feature
> Reporter: shubham pathak
> Assignee: shubham pathak
>
> Developing a ParquetReaderOperator which would allow apex users to read
> records from parquet files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
