[GitHub] [hive] shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
shameersss1 commented on a change in pull request #829: Hive JDBC Storage
Handler: Support For Writing Data to JDBC Data Source
URL: https://github.com/apache/hive/pull/829#discussion_r340521728
##
File path:
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
##
@@ -172,6 +177,67 @@ public int getTotalNumberOfRecords(Configuration conf)
throws HiveJdbcDatabaseAc
}
}
+ public RecordWriter getRecordWriter(TaskAttemptContext context)
+ throws IOException {
+Configuration conf = context.getConfiguration();
+Connection conn = null;
+PreparedStatement ps = null;
+String dbProductName =
conf.get(JdbcStorageConfig.DATABASE_TYPE.getPropertyName()).toUpperCase();
+String tableName = conf.get(JdbcStorageConfig.TABLE.getPropertyName());
+
+if (tableName == null || tableName.isEmpty()) {
+ throw new IllegalArgumentException("Table name should be defined");
+}
+
+String[] columnNames = conf.get(serdeConstants.LIST_COLUMNS).split(",");
+
+try {
+ initializeDatabaseConnection(conf);
+ conn = dbcpDataSource.getConnection();
+ ps = conn.prepareStatement(
+ constructQuery(tableName, columnNames, dbProductName));
+ return new org.apache.hadoop.mapreduce.lib.db.DBOutputFormat()
+ .new DBRecordWriter(conn, ps);
+} catch (Exception e) {
+ cleanupResources(conn, ps, null);
+ throw new IOException(e.getMessage());
+}
+ }
+
+ /**
+ * Constructs the query used as the prepared statement to insert data.
+ *
+ * @param table
+ * the table to insert into
+ * @param columnNames
+ * the columns to insert into.
+ * @param dbProductName
+ * type of database
+ *
+ */
+ public String constructQuery(String table, String[] columnNames, String
dbProductName) {
+if(columnNames == null) {
+ throw new IllegalArgumentException("Column names may not be null");
+}
+
+StringBuilder query = new StringBuilder();
+query.append("INSERT INTO ").append(table).append(" VALUES (");
+
+for (int i = 0; i < columnNames.length; i++) {
+ query.append("?");
+ if(i != columnNames.length - 1) {
+query.append(",");
+ }
+}
+
+if (!dbProductName.equals(DatabaseType.DERBY.toString()) &&
!dbProductName.equals(DatabaseType.ORACLE.toString())
Review comment:
> Specific logic for different database products should not be handled with
if clauses over here. Instead, you can create a method with default
implementation for most common case in this generic accessor and override
behavior if needed on a per-accessor basis (see e.g., `addLimitToQuery` method
for an example, which is overridden for Oracle accessor, MySQL, etc).
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
[GitHub] [hive] shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
shameersss1 commented on a change in pull request #829: Hive JDBC Storage
Handler: Support For Writing Data to JDBC Data Source
URL: https://github.com/apache/hive/pull/829#discussion_r340521562
##
File path:
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java
##
@@ -172,6 +177,67 @@ public int getTotalNumberOfRecords(Configuration conf)
throws HiveJdbcDatabaseAc
}
}
+ public RecordWriter getRecordWriter(TaskAttemptContext context)
+ throws IOException {
+Configuration conf = context.getConfiguration();
Review comment:
> L182-185 can be moved below first if condition.
Moved.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
[GitHub] [hive] shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source URL: https://github.com/apache/hive/pull/829#discussion_r340521488 ## File path: ql/src/test/results/clientpositive/llap/jdbc_handler.q.out ## @@ -118,6 +118,25 @@ POSTHOOK: type: QUERY POSTHOOK: Input: default@ext_simple_derby_table A masked pattern was here 200 +PREHOOK: query: insert into ext_simple_derby_table values(100) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +PREHOOK: Output: default@ext_simple_derby_table +POSTHOOK: query: insert into ext_simple_derby_table values(100) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +POSTHOOK: Output: default@ext_simple_derby_table +PREHOOK: query: select * from ext_simple_derby_table +PREHOOK: type: QUERY +PREHOOK: Input: default@ext_simple_derby_table + A masked pattern was here +POSTHOOK: query: select * from ext_simple_derby_table +POSTHOOK: type: QUERY +POSTHOOK: Input: default@ext_simple_derby_table + A masked pattern was here +20 +200 +100 PREHOOK: query: CREATE EXTERNAL TABLE tables Review comment: > Can we add additional testing? For instance: > > 1. Inserting rows with column values using all different Hive primitive types (boolean, tinyint, smallint, int, bigint, float, double, decimal, char, varchar, string, date, timestamp, etc.)? If any of those types are not supported, e.g., complex types, we already throw a user friendly error but please add a negative tests for those. > 2. Inserting multiple rows. > 3. CTAS statement. Added more test cases! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[GitHub] [hive] shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
shameersss1 commented on a change in pull request #829: Hive JDBC Storage
Handler: Support For Writing Data to JDBC Data Source
URL: https://github.com/apache/hive/pull/829#discussion_r340521259
##
File path:
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcRecordWriter.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.storage.jdbc;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.DBRecordWriter;
+import org.apache.hadoop.util.StringUtils;
+
+/**
+ * JdbcRecordWriter is wrapper class to write data to the underlying database.
+ */
+public class JdbcRecordWriter implements RecordWriter {
Review comment:
> We could make `JdbcRecordWriter` extend `DBRecordWriter`? If there are any
changes in this class behavior wrt `DBRecordWriter`, let's add comments to
document that.
we cannot extend DBRecordWriter since it is a non-static inner class of
DBOutputFormat
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
[GitHub] [hive] shameersss1 commented on a change in pull request #829: Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
shameersss1 commented on a change in pull request #829: Hive JDBC Storage
Handler: Support For Writing Data to JDBC Data Source
URL: https://github.com/apache/hive/pull/829#discussion_r340520616
##
File path:
jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/JdbcOutputFormat.java
##
@@ -18,19 +18,25 @@
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.shims.ShimLoader;
import org.apache.hadoop.io.MapWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.OutputFormat;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.util.Progressable;
+import org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor;
import java.io.IOException;
import java.util.Properties;
public class JdbcOutputFormat implements OutputFormat,
HiveOutputFormat {
+ private org.apache.hadoop.mapreduce.RecordWriter recordWriter;
Review comment:
> These two objects can be moved to `public RecordWriter
getHiveRecordWriter`, since they are not being accessed outside of the scope of
that method.
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
