[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin closed the pull request at:

https://github.com/apache/incubator-hawq/pull/972


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[edespino]

Github user edespino commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r102664824
  
--- Diff: pxf/build.gradle ---
@@ -431,6 +431,38 @@ project('pxf-hbase') {
 }
 }
 
+
+project('pxf-jdbc') {
+dependencies {
+compile(project(':pxf-api'))
+compile(project(':pxf-service'))
+compile "org.apache.hadoop:hadoop-common:$hadoopVersion"
+compile "org.apache.hadoop:hadoop-hdfs:$hadoopVersion"
+testCompile "mysql:mysql-connector-java:5.1.6"
+}
+tasks.withType(JavaCompile) {
+options.encoding = "UTF-8"
+}
+
+ospackage {
+packageName = versionedPackageName("${project.name}")
+summary = 'HAWQ Extension Framework (PXF), JDBC plugin'
+description = 'Querying external data stored in RelationDatabase 
using JDBC.'
--- End diff --

"RelationDatabase" should be "Relation Database"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r86278188
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,297 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
--- End diff --

At present  support of these three kinds of commonly used, the future can 
also increase other types.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[shivzone]

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r86221613
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[shivzone]

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r86204956
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,297 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+//30 days
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 1000);
+//365 days
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws UserDataException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
UserDataException {
+super(inConf);
+if 

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[shivzone]

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r86202298
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,297 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+//30 days
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 1000);
+//365 days
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);
--- End diff --

what is the factor of 30 ? You are converting a year to 365 days .. why 
apply a multiple of 30 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not 

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[shivzone]

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r86202717
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,297 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
--- End diff --

Please define private static final variables for each of these units eg: 
SECONDS_IN_MINUTE MINUTES_IN_HOUR HOURS_IN_DAY or to keep things simple just 
define MILLISECONDS_IN_DAY and the other 2 can be derived


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85867311
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/utils/ByteUtil.java 
---
@@ -0,0 +1,86 @@
+package org.apache.hawq.pxf.plugins.jdbc.utils;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+/**
+ * A tool class, used to deal with byte array merging, split and other 
methods.
+ */
+public class ByteUtil {
+
+public static byte[] mergeBytes(byte[] b1, byte[] b2) {
--- End diff --

This method is simple, I do not want to import a dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85867986
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/utils/ByteUtil.java 
---
@@ -0,0 +1,86 @@
+package org.apache.hawq.pxf.plugins.jdbc.utils;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+/**
+ * A tool class, used to deal with byte array merging, split and other 
methods.
+ */
+public class ByteUtil {
+
+public static byte[] mergeBytes(byte[] b1, byte[] b2) {
--- End diff --

This method is simple, I do not want to import a dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[sansanichfb]

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85825494
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,298 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+//30 days
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 1000);
+//365 days
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws UserDataException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
UserDataException  {
+super(inConf);
+

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85683989
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcMySqlExtensionTest.java
 ---
@@ -0,0 +1,303 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.sun.org.apache.xml.internal.utils.StringComparable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class JdbcMySqlExtensionTest {
+private static final Log LOG = 
LogFactory.getLog(JdbcMySqlExtensionTest.class);
+static String MYSQL_URL = "jdbc:mysql://localhost:3306/demodb";
--- End diff --

I renamed JdbcMySqlExtensionTest to SqlBuilderTest.
 Validate SQL string generated by the  
JdbcPartitionFragmenter.buildFragmenterSql method and the 
WhereSQLBuilder.buildWhereSQL method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85678810
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcMySqlExtensionTest.java
 ---
@@ -0,0 +1,303 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.sun.org.apache.xml.internal.utils.StringComparable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class JdbcMySqlExtensionTest {
+private static final Log LOG = 
LogFactory.getLog(JdbcMySqlExtensionTest.class);
+static String MYSQL_URL = "jdbc:mysql://localhost:3306/demodb";
--- End diff --

I think that the sql builder logic can be tested separately as part of the 
WhereSQLBuilder tests.
If you still want to keep the test, one option is to add it to the 
automation tests of PXF, which as far as I know were not open sourced yet 
(maybe this is a good reason to do that :)).
Another option is to make it a separate module - not part of the unit 
tests, and not something that is running as part of the compilation. Maybe wait 
for other people to give their opinion on that matter...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85678674
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilderTest.java
 ---
@@ -0,0 +1,81 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser.LogicalOperation;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.junit.Test;
+
+import static org.apache.hawq.pxf.api.FilterParser.Operation.*;
+import static org.junit.Assert.assertEquals;
+
+public class JdbcFilterBuilderTest {
+@Test
+public void parseFilterWithThreeOperations() throws Exception {
+//orgin sql => cdate>'2008-02-01' and cdate<'2008-12-01' and amt > 
1200
+//filterstr="a1c\"first\"o5a2c2o2l0";//col_1=first and col_2=2
+String filterstr = 
"a1c\"2008-02-01\"o2a1c\"2008-12-01\"o1l0a2c1200o2l1"; //col_1>'first' and 
col_1<'2008-12-01' or col_2 > 1200;
+JdbcFilterBuilder builder = new JdbcFilterBuilder();
+
+LogicalFilter filterList = (LogicalFilter) 
builder.getFilterObject(filterstr);
+assertEquals(LogicalOperation.HDOP_OR, filterList.getOperator());
+LogicalFilter l1_left = (LogicalFilter) 
filterList.getFilterList().get(0);
+BasicFilter l1_right = (BasicFilter) 
filterList.getFilterList().get(1);
+//column_2 > 1200
+assertEquals(2, l1_right.getColumn().index());
+assertEquals(HDOP_GT, l1_right.getOperation());
+assertEquals(1200L, l1_right.getConstant().constant());
+
+assertEquals(LogicalOperation.HDOP_AND, l1_left.getOperator());
+BasicFilter l2_left = (BasicFilter) l1_left.getFilterList().get(0);
+BasicFilter l2_right = (BasicFilter) 
l1_left.getFilterList().get(1);
+
+//column_1 > '2008-02-01'
+assertEquals(1, l2_left.getColumn().index());
+assertEquals(HDOP_GT, l2_left.getOperation());
+assertEquals("2008-02-01", l2_left.getConstant().constant());
+
+//column_2 = 5
+assertEquals(1, l2_right.getColumn().index());
+assertEquals(HDOP_LT, l2_right.getOperation());
+assertEquals("2008-12-01", l2_right.getConstant().constant());
+
+}
+
+@Test
+public void parseFilterWithLogicalOperation() throws Exception {
+WhereSQLBuilder builder = new WhereSQLBuilder(null);
--- End diff --

Since `buildWhereSQL()` is a public function, it should be easy to test it 
- create a `WhereSQLBuilder` instance like you do here, mock the input to have 
a filter string property, and check that calling this function returns the 
expected sql string.
That way the test won't rely on any outsider resources, and errors in the 
logic will be easy to identify.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85678597
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
--- End diff --

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85678564
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+return;
  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85677162
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcMySqlExtensionTest.java
 ---
@@ -0,0 +1,303 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.sun.org.apache.xml.internal.utils.StringComparable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class JdbcMySqlExtensionTest {
+private static final Log LOG = 
LogFactory.getLog(JdbcMySqlExtensionTest.class);
+static String MYSQL_URL = "jdbc:mysql://localhost:3306/demodb";
--- End diff --

This `test` is used to test the correctness of sql, it is important.
If want to keep it in the project, how to do?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85677094
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilderTest.java
 ---
@@ -0,0 +1,81 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser.LogicalOperation;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.junit.Test;
+
+import static org.apache.hawq.pxf.api.FilterParser.Operation.*;
+import static org.junit.Assert.assertEquals;
+
+public class JdbcFilterBuilderTest {
+@Test
+public void parseFilterWithThreeOperations() throws Exception {
+//orgin sql => cdate>'2008-02-01' and cdate<'2008-12-01' and amt > 
1200
+//filterstr="a1c\"first\"o5a2c2o2l0";//col_1=first and col_2=2
+String filterstr = 
"a1c\"2008-02-01\"o2a1c\"2008-12-01\"o1l0a2c1200o2l1"; //col_1>'first' and 
col_1<'2008-12-01' or col_2 > 1200;
+JdbcFilterBuilder builder = new JdbcFilterBuilder();
+
+LogicalFilter filterList = (LogicalFilter) 
builder.getFilterObject(filterstr);
+assertEquals(LogicalOperation.HDOP_OR, filterList.getOperator());
+LogicalFilter l1_left = (LogicalFilter) 
filterList.getFilterList().get(0);
+BasicFilter l1_right = (BasicFilter) 
filterList.getFilterList().get(1);
+//column_2 > 1200
+assertEquals(2, l1_right.getColumn().index());
+assertEquals(HDOP_GT, l1_right.getOperation());
+assertEquals(1200L, l1_right.getConstant().constant());
+
+assertEquals(LogicalOperation.HDOP_AND, l1_left.getOperator());
+BasicFilter l2_left = (BasicFilter) l1_left.getFilterList().get(0);
+BasicFilter l2_right = (BasicFilter) 
l1_left.getFilterList().get(1);
+
+//column_1 > '2008-02-01'
+assertEquals(1, l2_left.getColumn().index());
+assertEquals(HDOP_GT, l2_left.getOperation());
+assertEquals("2008-02-01", l2_left.getConstant().constant());
+
+//column_2 = 5
+assertEquals(1, l2_right.getColumn().index());
+assertEquals(HDOP_LT, l2_right.getOperation());
+assertEquals("2008-12-01", l2_right.getConstant().constant());
+
+}
+
+@Test
+public void parseFilterWithLogicalOperation() throws Exception {
+WhereSQLBuilder builder = new WhereSQLBuilder(null);
--- End diff --

WhereSQLBuilder used to build sql statement, its 'test' through 
JdbcMySqlExtensionTest completed.
There are other ways to test the correctness of sql statement?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85676673
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85476012
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+return;
  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85476340
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
--- End diff --

  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85473708
  
--- Diff: pxf/pxf-jdbc/src/test/resources/core-site.xml ---
@@ -0,0 +1,28 @@
+
--- End diff --

The test/resource directory contains information about specific setup 
(hadoop & mysql), and should probably not be included here.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85473491
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenterTest.java
 ---
@@ -0,0 +1,208 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.junit.Test;
+
+import java.text.ParseException;
+import java.util.Calendar;
+import java.util.List;
+
+import static org.junit.Assert.*;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+
+public class JdbcPartitionFragmenterTest {
+InputData inputData;
+
+@Test
+public void testPartionByDate() throws Exception {
+prepareConstruction();
+when(inputData.getDataSource()).thenReturn("sales");
+
when(inputData.getUserProperty("PARTITION_BY")).thenReturn("cdate:date");
+
when(inputData.getUserProperty("RANGE")).thenReturn("2008-01-01:2009-01-01");
+when(inputData.getUserProperty("INTERVAL")).thenReturn("1:month");
+
+JdbcPartitionFragmenter fragment = new 
JdbcPartitionFragmenter(inputData);
+List fragments = fragment.getFragments();
+assertEquals(fragments.size(), 12);
+
+//fragment - 1
+byte[] frag_meta = fragments.get(0).getMetadata();
+byte[][] newb = ByteUtil.splitBytes(frag_meta, 8);
+long frag_start = ByteUtil.toLong(newb[0]);
+long frag_end = ByteUtil.toLong(newb[1]);
+assertDateEquals(frag_start, 2008, 1, 1);
+assertDateEquals(frag_end, 2008, 2, 1);
+
+//fragment - 12
+frag_meta = fragments.get(11).getMetadata();
+newb = ByteUtil.splitBytes(frag_meta, 8);
+frag_start = ByteUtil.toLong(newb[0]);
+frag_end = ByteUtil.toLong(newb[1]);
+assertDateEquals(frag_start, 2008, 12, 1);
+assertDateEquals(frag_end, 2009, 1, 1);
+
+//when end_date > start_date
+
when(inputData.getUserProperty("RANGE")).thenReturn("2008-01-01:2001-01-01");
+fragment = new JdbcPartitionFragmenter(inputData);
+assertEquals(0,fragment.getFragments().size());
+}
+
+@Test
+public void testPartionByDate2() throws Exception {
--- End diff --

typo Partition


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85473433
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcMySqlExtensionTest.java
 ---
@@ -0,0 +1,303 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.sun.org.apache.xml.internal.utils.StringComparable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class JdbcMySqlExtensionTest {
+private static final Log LOG = 
LogFactory.getLog(JdbcMySqlExtensionTest.class);
+static String MYSQL_URL = "jdbc:mysql://localhost:3306/demodb";
+InputData inputData;
+
+@Before
+public void setup() throws Exception {
+LOG.info("JdbcMySqlExtensionTest.setup()");
+prepareConstruction();
+JdbcWriter writer = new JdbcWriter(inputData);
+writer.openForWrite();
+
+//create table
+writer.executeSQL("CREATE TABLE sales (id int primary key, cdate 
date, amt decimal(10,2),grade varchar(30))");
+//INSERT DEMO data
+String[] inserts = {"INSERT INTO sales values (1, 
DATE('2008-01-01'), 1000,'general')",
+"INSERT INTO sales values (2, DATE('2008-02-01'), 
900,'bad')",
+"INSERT INTO sales values (3, DATE('2008-03-10'), 
1200,'good')",
+"INSERT INTO sales values (4, DATE('2008-04-10'), 
1100,'good')",
+"INSERT INTO sales values (5, DATE('2008-05-01'), 
1010,'general')",
+"INSERT INTO sales values (6, DATE('2008-06-01'), 
850,'bad')",
+"INSERT INTO sales values (7, DATE('2008-07-01'), 
1400,'excellent')",
+"INSERT INTO sales values (8, DATE('2008-08-01'), 
1500,'excellent')",
+"INSERT INTO sales values (9, DATE('2008-09-01'), 
1000,'good')",
+"INSERT INTO sales values (10, DATE('2008-10-01'), 
800,'bad')",
+"INSERT INTO sales values (11, DATE('2008-11-01'), 
1250,'good')",
+"INSERT INTO sales values (12, DATE('2008-12-01'), 
1300,'excellent')",
+"INSERT INTO sales values (15, DATE('2009-01-01'), 
1500,'excellent')",
+"INSERT INTO sales values (16, DATE('2009-02-01'), 
1340,'excellent')",
+"INSERT INTO sales values (13, DATE('2009-03-01'), 
1250,'good')",
+"INSERT INTO sales values (14, DATE('2009-04-01'), 
1300,'excellent')"};
+for (String sql : inserts)
+writer.executeSQL(sql);
+
+writer.closeWrite();
+}
+
+@After
+public void cleanup() throws Exception {
+LOG.info("JdbcMySqlExtensionTest.cleanup()");
+prepareConstruction();
+JdbcWriter writer = new JdbcWriter(inputData);
+writer.openForWrite();
+writer.executeSQL("drop table sales");
+writer.closeWrite();
+}
+
+@Test
+public void testIdFilter() throws Exception {
+prepareConstruction();
+when(inputData.hasFilter()).thenReturn(true);
+

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85473415
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcMySqlExtensionTest.java
 ---
@@ -0,0 +1,303 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.sun.org.apache.xml.internal.utils.StringComparable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class JdbcMySqlExtensionTest {
+private static final Log LOG = 
LogFactory.getLog(JdbcMySqlExtensionTest.class);
+static String MYSQL_URL = "jdbc:mysql://localhost:3306/demodb";
--- End diff --

does this test require a mysql instance running locally? if so, it cannot 
be here as part of the unit-tests, but rather should be moved to the automation 
suite. The unit tests have to be self contained.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85473219
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilderTest.java
 ---
@@ -0,0 +1,81 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser.LogicalOperation;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.junit.Test;
+
+import static org.apache.hawq.pxf.api.FilterParser.Operation.*;
+import static org.junit.Assert.assertEquals;
+
+public class JdbcFilterBuilderTest {
+@Test
+public void parseFilterWithThreeOperations() throws Exception {
+//orgin sql => cdate>'2008-02-01' and cdate<'2008-12-01' and amt > 
1200
+//filterstr="a1c\"first\"o5a2c2o2l0";//col_1=first and col_2=2
+String filterstr = 
"a1c\"2008-02-01\"o2a1c\"2008-12-01\"o1l0a2c1200o2l1"; //col_1>'first' and 
col_1<'2008-12-01' or col_2 > 1200;
+JdbcFilterBuilder builder = new JdbcFilterBuilder();
+
+LogicalFilter filterList = (LogicalFilter) 
builder.getFilterObject(filterstr);
+assertEquals(LogicalOperation.HDOP_OR, filterList.getOperator());
+LogicalFilter l1_left = (LogicalFilter) 
filterList.getFilterList().get(0);
+BasicFilter l1_right = (BasicFilter) 
filterList.getFilterList().get(1);
+//column_2 > 1200
+assertEquals(2, l1_right.getColumn().index());
+assertEquals(HDOP_GT, l1_right.getOperation());
+assertEquals(1200L, l1_right.getConstant().constant());
+
+assertEquals(LogicalOperation.HDOP_AND, l1_left.getOperator());
+BasicFilter l2_left = (BasicFilter) l1_left.getFilterList().get(0);
+BasicFilter l2_right = (BasicFilter) 
l1_left.getFilterList().get(1);
+
+//column_1 > '2008-02-01'
+assertEquals(1, l2_left.getColumn().index());
+assertEquals(HDOP_GT, l2_left.getOperation());
+assertEquals("2008-02-01", l2_left.getConstant().constant());
+
+//column_2 = 5
+assertEquals(1, l2_right.getColumn().index());
+assertEquals(HDOP_LT, l2_right.getOperation());
+assertEquals("2008-12-01", l2_right.getConstant().constant());
+
+}
+
+@Test
+public void parseFilterWithLogicalOperation() throws Exception {
+WhereSQLBuilder builder = new WhereSQLBuilder(null);
--- End diff --

could you add tests for `WhereSQLBuilder.buildWhereSQL()`, not just for the 
parsing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85472961
  
--- Diff: 
pxf/pxf-jdbc/src/test/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilderTest.java
 ---
@@ -0,0 +1,81 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser.LogicalOperation;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.junit.Test;
+
+import static org.apache.hawq.pxf.api.FilterParser.Operation.*;
+import static org.junit.Assert.assertEquals;
+
+public class JdbcFilterBuilderTest {
+@Test
+public void parseFilterWithThreeOperations() throws Exception {
+//orgin sql => cdate>'2008-02-01' and cdate<'2008-12-01' and amt > 
1200
+//filterstr="a1c\"first\"o5a2c2o2l0";//col_1=first and col_2=2
+String filterstr = 
"a1c\"2008-02-01\"o2a1c\"2008-12-01\"o1l0a2c1200o2l1"; //col_1>'first' and 
col_1<'2008-12-01' or col_2 > 1200;
+JdbcFilterBuilder builder = new JdbcFilterBuilder();
+
+LogicalFilter filterList = (LogicalFilter) 
builder.getFilterObject(filterstr);
+assertEquals(LogicalOperation.HDOP_OR, filterList.getOperator());
+LogicalFilter l1_left = (LogicalFilter) 
filterList.getFilterList().get(0);
+BasicFilter l1_right = (BasicFilter) 
filterList.getFilterList().get(1);
+//column_2 > 1200
+assertEquals(2, l1_right.getColumn().index());
+assertEquals(HDOP_GT, l1_right.getOperation());
+assertEquals(1200L, l1_right.getConstant().constant());
+
+assertEquals(LogicalOperation.HDOP_AND, l1_left.getOperator());
+BasicFilter l2_left = (BasicFilter) l1_left.getFilterList().get(0);
+BasicFilter l2_right = (BasicFilter) 
l1_left.getFilterList().get(1);
+
+//column_1 > '2008-02-01'
+assertEquals(1, l2_left.getColumn().index());
+assertEquals(HDOP_GT, l2_left.getOperation());
+assertEquals("2008-02-01", l2_left.getConstant().constant());
+
+//column_2 = 5
--- End diff --

also this comment


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85472352
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/utils/DbProduct.java
 ---
@@ -0,0 +1,51 @@
+package org.apache.hawq.pxf.plugins.jdbc.utils;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.UnsupportedTypeException;
+import org.iq80.leveldb.DB;
--- End diff --

licensing question - do we need to add this resource to the licenses list?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85472163
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/WhereSQLBuilder.java
 ---
@@ -0,0 +1,140 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+/**
+ * Parse filter object generated by parent class  {@link 
org.apache.hawq.pxf.plugins.jdbc.JdbcFilterBuilder},
+ * and build WHERE statement.
+ * For Multiple filters , currently only support HDOP_AND .
+ * The unsupported Filter operation and  LogicalOperation ,will return 
null statement.
+ *
+ */
+public class WhereSQLBuilder extends JdbcFilterBuilder {
+private InputData inputData;
+
+public WhereSQLBuilder(InputData input) {
+inputData = input;
+}
+
+/**
+ * 1.check for LogicalOperator, Jdbc currently only support HDOP_AND.
+ * 2.and convert to BasicFilter List.
+ */
+private static List convertBasicFilterList(Object filter, 
List returnList) throws UnsupportedFilterException {
+if (returnList == null)
+returnList = new ArrayList<>();
+if (filter instanceof BasicFilter) {
+returnList.add((BasicFilter) filter);
+return returnList;
+}
+LogicalFilter lfilter = (LogicalFilter) filter;
+if (lfilter.getOperator() != 
FilterParser.LogicalOperation.HDOP_AND)
+throw new UnsupportedFilterException("unsupported 
LogicalOperation : " + lfilter.getOperator());
+for (Object f : lfilter.getFilterList()) {
+returnList = convertBasicFilterList(f, returnList);
+}
+return returnList;
+}
+
+public String buildWhereSQL(String db_product) throws Exception {
+if (!inputData.hasFilter()) return null;
+List filters = null;
+try {
+String filterString = inputData.getFilterString();
+Object filterobj = getFilterObject(filterString);
+
+filters = convertBasicFilterList(filterobj, filters);
+StringBuffer sb = new StringBuffer("1=1");
+for (Object obj : filters) {
+BasicFilter filter = (BasicFilter) obj;
+sb.append(" AND ");
+
+ColumnDescriptor column = 
inputData.getColumn(filter.getColumn().index());
+//the column name of filter
+sb.append(column.columnName());
+
+//the operation of filter
+FilterParser.Operation op = filter.getOperation();
+switch (op) {
+case HDOP_LT:
+sb.append("<");
+break;
+case HDOP_GT:
+sb.append(">");
+break;
+case HDOP_LE:
+sb.append("<=");
+break;
+case HDOP_GE:
+sb.append(">=");
+break;
+case HDOP_EQ:
+sb.append("=");
+break;
+default:
+throw new UnsupportedFilterException("unsupported 
Filter operation : " + op);
+}
+//
+DbProduct dbProduct = 

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85471996
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcReadResolver.java
 ---
@@ -0,0 +1,116 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.*;
+import org.apache.hawq.pxf.api.io.DataType;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.sql.ResultSet;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Class JdbcReadResolver Read the Jdbc ResultSet, and generates the data 
type - List .
+ */
+public class JdbcReadResolver extends Plugin implements ReadResolver {
+private static final Log LOG = 
LogFactory.getLog(JdbcReadResolver.class);
+//HAWQ Table column definitions
+private ArrayList columns = null;
+
+public JdbcReadResolver(InputData input) {
+super(input);
+columns = input.getTupleDescription();
+}
+
+@Override
+public List getFields(OneRow row) throws Exception {
+//LOG.info("getFields");
--- End diff --

please remove all commented code in this function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85471780
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPlugin.java ---
@@ -0,0 +1,114 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+
+import java.sql.*;
+
+/**
+ * This class resolves the jdbc connection parameter and manages the 
opening and closing of the jdbc connection.
+ * Implemented subclasses: {@link JdbcReadAccessor}.
+ *
+ */
+public class JdbcPlugin extends Plugin {
+private static final Log LOG = LogFactory.getLog(JdbcPlugin.class);
+
+//jdbc connection parameters
+protected String jdbcDriver = null;
+protected String dbUrl = null;
+protected String user = null;
+protected String pass = null;
+protected String tblName = null;
+protected int batchSize = 100;
+
+//jdbc connection
+protected Connection dbconn = null;
+//database type，from DatabaseMetaData.getDatabaseProductName()
+protected String dbProduct = null;
+
+/**
+ * parse
+ *
+ * @param input the input data
+ */
+public JdbcPlugin(InputData input) {
+super(input);
+jdbcDriver = input.getUserProperty("JDBC_DRIVER");
+dbUrl = input.getUserProperty("DB_URL");
+//dbUrl = "jdbc:mysql://192.168.200.6:3306/demodb";
--- End diff --

please remove commented code, or edit to be an example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85471415
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+return;
  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85471130
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+return;
  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85470831
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
--- End diff --

Please change all variables to be camelCase, for example `partitionBy` 
instead of `partition_by`.
(The coding conventions can be found here 
https://cwiki.apache.org/confluence/display/HAWQ/PXF+Coding+Conventions#PXFCodingConventions-Naming)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85470541
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 * 
1000);//365 day
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws JdbcFragmentException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws 
JdbcFragmentException {
+super(inConf);
+if(inConf.getUserProperty("PARTITION_BY") == null )
+return;
  

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85470195
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of 
transforming
+ * an input data path  (an JDBC Database table name  and user request 
parameters)  into a list of regions
+ * that belong to this table.
+ * 
+ * The parameter Patterns 
+ * There are three  parameters,  the format is as follows:
+ * 
+ * 
PARTITION_BY=column_name:column_type=start_value[:end_value]=interval_num[:interval_unit]
+ * 
+ * The PARTITION_BY parameter can be split by colon(':'),the 
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'. 
+ * The RANGE parameter can be split by colon(':') ,used to 
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value' 
.If the column_type is int,
+ * the end_value can be empty. If the 
column_typeis enum,the parameter RANGE 
can be empty. 
+ * The INTERVAL parameter can be split by colon(':'), 
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must 
be split by colon, and interval_unit can be 
year,month,day.
+ * When column_type is int, the 
interval_unit can be empty.
+ * When column_type is enum,the 
INTERVAL parameter can be empty.
+ * 
+ * 
+ * The syntax examples is :
+ * 
PARTITION_BY=createdate:date=2008-01-01:2010-01-01=1:month'
 
+ * PARTITION_BY=year:int=2008:2010=1 
+ * PARTITION_BY=grade:enum=excellent:good:general:bad
+ * 
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partition_by = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the 
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 
1000);//30 day
--- End diff --

minor - please add a space before the comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85470171
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
 ---
@@ -0,0 +1,284 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+
+/**
+ * Fragmenter class for JDBC data resources.
--- End diff --

Great documentation! thanks for providing the examples as well.
minor - could you please remove the double spaces?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85470036
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilder.java
 ---
@@ -0,0 +1,145 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.LogicalFilter;
+
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Uses the filter parser code to build a filter object, either simple - a
+ * single {@link BasicFilter} object or a
+ * compound - a {@link java.util.List} of
+ * {@link BasicFilter} objects.
+ * The subclass {@link org.apache.hawq.pxf.plugins.jdbc.WhereSQLBuilder} 
will use the filter for
+ * generate WHERE statement.
+ */
+public class JdbcFilterBuilder implements FilterParser.FilterBuilder  {
--- End diff --

this class looks very similar to `HiveFilterBuilder.java` - maybe consider 
inheriting from it or creating a common implementation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[hornn]

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/972#discussion_r85469744
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcFilterBuilder.java
 ---
@@ -0,0 +1,145 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.LogicalFilter;
+
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Uses the filter parser code to build a filter object, either simple - a
+ * single {@link BasicFilter} object or a
+ * compound - a {@link java.util.List} of
+ * {@link BasicFilter} objects.
+ * The subclass {@link org.apache.hawq.pxf.plugins.jdbc.WhereSQLBuilder} 
will use the filter for
+ * generate WHERE statement.
+ */
+public class JdbcFilterBuilder implements FilterParser.FilterBuilder  {
+/**
+ * Translates a filterString into a {@link BasicFilter} or a
+ * list of such filters.
+ *
+ * @param filterString the string representation of the filter
+ * @return a single {@link BasicFilter}
+ * object or a {@link java.util.List} of
+ * {@link BasicFilter} objects.
+ * @throws Exception if parsing the filter failed or filter is not a 
basic
+ * filter or list of basic filters
+ */
+public Object getFilterObject(String filterString) throws Exception {
+// First check for LogicalOperator, Jdbc currently only support 
HDOP_AND.
+  //  if (filterString.contains("l1") || filterString.contains("l2"))
--- End diff --

please remove commented code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin

[jiadexin]

GitHub user jiadexin opened a pull request:

https://github.com/apache/incubator-hawq/pull/972

HAWQ-1108 Add JDBC PXF Plugin

The PXF JDBC plug-in reads data stored in Traditional relational 
database,ie : mysql,ORACLE,postgresql.
For more information, please refer 
to:https://github.com/inspur-insight/incubator-hawq/blob/HAWQ-1108/pxf/pxf-jdbc/README.md
 .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/inspur-insight/incubator-hawq HAWQ-1108

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/972.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #972


commit 2cc75e672d01926ef97d7c50485f4979d4866b3c
Author: Devin Jia 
Date:   2016-10-18T08:08:50Z

Merge pull request #1 from apache/master

re fork

commit 10f68af5ade550b6c24abe371fff4a40349829b3
Author: Devin Jia 
Date:   2016-10-25T06:31:07Z

the first commit

commit 5a814211ecf8f8f1e7d1487bdda33c3b72f1b990
Author: Devin Jia 
Date:   2016-10-25T07:10:12Z

modify parent pxf build.gradle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---