DBStorage is not working with other storage in pig script. means DBStorage
is not working with multiple storage statement.
What I was trying for: 1) I was trying to Store one output using DBStorage
And was trying to store same or different output using simple Store to file
system 2) I also tried to store using DBStorage and using my custom store
function
But in both cases it not storing the data to database. If I comment out
another storage than DBStorage is working properly.
Even its not throwing any exception or error on reducer's machine..
Can anyone point out the problem?
DBStorage is not working with Simple Store to file system. Its only working
if I put only DBStorage no other Store Statement..
pv_by_industry = GROUP profile_view by viewee_industry_id
pv_avg_by_industry = FOREACH pv_by_industry GENERATE
group as viewee_industry_id, AVG(profie_view) AS average_pv;
STORE pv_avg_by_industry INTO '/tmp/hardik';
STORE pv_avg_by_industry into /tmp/hardik/db' INTO
DBStorage('com.mysql.jdbc.Driver',
'dbc:mysql://hostname/dbname', 'user',
'pass',
'INSERT INTO table (viewee_industry_id,average_pv) VALUES(?,?)');
Few things are came into picture when I was debugging it.
DBStorage is setting Auto commit to False.
So when the batch is executed it will not be auto committed.
After executing batch OutputCommiter's method commitTask in DBStorage
(inline class' method) was called in which commit is written
if (ps != null) {
try {System.out.println("Executing Batch in commitTask");
ps.executeBatch();
con.commit();
ps.close();
con.close();
ps = null;
con = null;
} catch (SQLException e) {System.out.println("Exception in
commitTask");
log.error("ps.close", e);
throw new IOException("JDBC Error", e);
}
and this method is called by PigOutputCommiter
public void commitTask(TaskAttemptContext context) throws IOException {
if(HadoopShims.isMap(context.getTaskAttemptID())) {
for (Pair<OutputCommitter, POStore> mapCommitter :
mapOutputCommitters) {
if (mapCommitter.first!=null) {
TaskAttemptContext updatedContext =
setUpContext(context,
mapCommitter.second);
mapCommitter.first.commitTask(updatedContext);
}
}
} else {
for (Pair<OutputCommitter, POStore> reduceCommitter :
reduceOutputCommitters) {
if (reduceCommitter.first!=null) {
TaskAttemptContext updatedContext =
setUpContext(context,
reduceCommitter.second);
reduceCommitter.first.commitTask(updatedContext);
}
}
}
}
But when this commitTask is called its connection and preparedStatment
Objects become null... so it ll not commit so data is not available in
Database.....
But if you write only DBStorage without any other Store statement in script
it will work properly..
Any clues???
--
Regards,
Hardik Shah.
