[
https://issues.apache.org/jira/browse/HIVE-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747110#action_12747110
]
Luis Ramos commented on HIVE-645:
---------------------------------
Edward,
Adding the jar got rid of the Failed for ExecDriver, thanks, HOWEVER, I still
don't see any difference on mysql (it doesn't insert or anything).
I did these steps:
1. add jar build/contrib/hive_contrib.jar;
2. CREATE TEMPORARY FUNCTION dboutput AS
'org.apache.hadoop.hive.contrib.genericudf.example.GenericUDFDBOutput';
3. SELECT
dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT INTO
status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM
web_logs GROUP BY day, httpstatus;
On my hive.log I see:
WARN mapred.JobClient (JobClient.java:configureCommandLineOptions(510)) - Use
GenericOptionsParser for parsing the arguments. Applications should implement
Tool for the same.
Output to console is:
$ hive -hiveconf hive.root.logger=INFO,console
hive> SELECT
dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT INTO
status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM
web_logs GROUP BY day, httpstatus;
09/08/24 22:21:00 INFO parse.ParseDriver: Parsing command: SELECT
dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT INTO
status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM
web_logs GROUP BY day, httpstatus
09/08/24 22:21:00 INFO parse.ParseDriver: Parse Completed
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed phase 1 of Semantic
Analysis
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/08/24 22:21:00 INFO metastore.HiveMetaStore: 0: get_table : db=default
tbl=web_logs
09/08/24 22:21:00 INFO metastore.HiveMetaStore: 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/08/24 22:21:00 INFO metastore.ObjectStore: ObjectStore, initialize called
09/08/24 22:21:00 INFO metastore.ObjectStore: Initialized ObjectStore
09/08/24 22:21:00 INFO hive.log: DDL: struct web_logs { string day, string
time, string url, string httpstatus}
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Get metadata for destination
tables
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed getting MetaData in
Semantic Analysis
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for FS(34)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for SEL(33)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for GBY(32)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for RS(31)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for GBY(30)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for SEL(29)
09/08/24 22:21:00 INFO ppd.OpProcFactory: Processing for TS(28)
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed partition pruning
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed sample pruning
09/08/24 22:21:00 INFO parse.SemanticAnalyzer: Completed plan generation
09/08/24 22:21:00 INFO ql.Driver: Semantic Analysis Completed
09/08/24 22:21:00 INFO ql.Driver: Starting command: SELECT
dboutput('jdbc:mysql://localhost:3306/analytics','root','password','INSERT INTO
status_count(date,status,count) VALUES (?,?,?)',day,httpstatus,count(1)) FROM
web_logs GROUP BY day, httpstatus
Total MapReduce jobs = 1
09/08/24 22:21:00 INFO ql.Driver: Total MapReduce jobs = 1
09/08/24 22:21:00 INFO exec.ExecDriver: BytesPerReducer=1000000000
maxReducers=999 totalInputFileSize=108
Number of reduce tasks not specified. Estimated from input data size: 1
09/08/24 22:21:00 INFO exec.ExecDriver: Number of reduce tasks not specified.
Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
09/08/24 22:21:00 INFO exec.ExecDriver: In order to change the average load for
a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
09/08/24 22:21:00 INFO exec.ExecDriver: set
hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
09/08/24 22:21:00 INFO exec.ExecDriver: In order to limit the maximum number of
reducers:
set hive.exec.reducers.max=<number>
09/08/24 22:21:00 INFO exec.ExecDriver: set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
09/08/24 22:21:00 INFO exec.ExecDriver: In order to set a constant number of
reducers:
set mapred.reduce.tasks=<number>
09/08/24 22:21:00 INFO exec.ExecDriver: set mapred.reduce.tasks=<number>
09/08/24 22:21:00 INFO exec.ExecDriver: adding libjars:
file:///home/hadoop/hive/build/contrib/hive_contrib.jar
09/08/24 22:21:00 INFO exec.ExecDriver: Processing alias web_logs
09/08/24 22:21:00 INFO exec.ExecDriver: Adding input file
hdfs://verizonmaster:50120/user/hive/warehouse/web_logs
09/08/24 22:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
09/08/24 22:21:00 INFO mapred.FileInputFormat: Total input paths to process : 1
09/08/24 22:21:00 INFO mapred.FileInputFormat: Total input paths to process : 1
Starting Job = job_200908201702_0102, Tracking URL =
http://verizonmaster:50030/jobdetails.jsp?jobid=job_200908201702_0102
09/08/24 22:21:01 INFO exec.ExecDriver: Starting Job = job_200908201702_0102,
Tracking URL =
http://verizonmaster:50030/jobdetails.jsp?jobid=job_200908201702_0102
Kill Command = /home/hadoop/hadoop-0.18.3/bin/../bin/hadoop job
-Dmapred.job.tracker=verizonmaster:50121 -kill job_200908201702_0102
09/08/24 22:21:01 INFO exec.ExecDriver: Kill Command =
/home/hadoop/hadoop-0.18.3/bin/../bin/hadoop job
-Dmapred.job.tracker=verizonmaster:50121 -kill job_200908201702_0102
2009-08-24 10:21:02,237 map = 0%, reduce = 0%
09/08/24 22:21:02 INFO exec.ExecDriver: 2009-08-24 10:21:02,237 map = 0%,
reduce = 0%
2009-08-24 10:21:08,253 map = 50%, reduce = 50%
09/08/24 22:21:08 INFO exec.ExecDriver: 2009-08-24 10:21:08,253 map = 50%,
reduce = 50%
2009-08-24 10:21:10,265 map = 100%, reduce = 100%
09/08/24 22:21:10 INFO exec.ExecDriver: 2009-08-24 10:21:10,265 map = 100%,
reduce = 100%
Ended Job = job_200908201702_0102
09/08/24 22:21:21 INFO exec.ExecDriver: Ended Job = job_200908201702_0102
09/08/24 22:21:21 INFO exec.FileSinkOperator: Moving tmp dir:
hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001 to:
hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001.intermediate
09/08/24 22:21:21 INFO exec.FileSinkOperator: Moving tmp dir:
hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/_tmp.10001.intermediate
to: hdfs://verizonmaster:50120/tmp/hive-hadoop/942129758/10001
OK
09/08/24 22:21:21 INFO ql.Driver: OK
09/08/24 22:21:21 INFO mapred.FileInputFormat: Total input paths to process : 1
09/08/24 22:21:21 INFO mapred.FileInputFormat: Total input paths to process : 1
1
1
Time taken: 21.03 seconds
I'm going to try other test cases and different data, I tried sending constants
into status_count but that didn't work either. Is there other jar's I might be
missing.
Thank You.
> A UDF that can export data to JDBC databases.
> ---------------------------------------------
>
> Key: HIVE-645
> URL: https://issues.apache.org/jira/browse/HIVE-645
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Contrib
> Reporter: Edward Capriolo
> Assignee: Edward Capriolo
> Priority: Minor
> Attachments: hive-645-2.patch, hive-645-3.patch, hive-645-4.patch,
> hive-645-5.patch, hive-645.patch
>
>
> A UDF that can export data to JDBC databases.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
