Hive streaming data ingest api Invalid method name: 'get_all_functions'

[Eva Donaldson]

I am having the worst time of tracking this down and figured it made it 
officially time to join the real club!


I am using the Hive Streaming Data Ingest API as detailed 
herehttps://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest.  On 
beginning the new batch when it gets to 
org.apache.hadoop.hive.ql.session.SessionState.start at the line that reads 
Hive.get(new HiveConf(startSs.sessionConf)).getMSC(); it kicks the following 
error:

  1.  org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.thrift.TApplicationException: Invalid method name: 
'get_all_functions'

In order to be able to see my version numbers my Maven declarations are as 
follows:

  1.      <dependency>
  2.          <groupId>org.apache.hive.hcatalog</groupId>
  3.          <artifactId>hive-hcatalog-streaming</artifactId>
  4.          <version>2.1.0</version>
  5.      </dependency>
  6.      <dependency>
  7.          <groupId>org.apache.hive.hcatalog</groupId>
  8.          <artifactId>hive-hcatalog-core</artifactId>
  9.          <version>2.1.0</version>
  10.     </dependency>
  11.     <dependency>
  12.         <groupId>org.apache.hive</groupId>
  13.         <artifactId>hive-jdbc</artifactId>
  14.         <version>2.1.0</version>
  15.     </dependency>

I declare my connection as follows

  1.  public HiveProcess(String dbname, String tablename, List<String> 
partitionVals)
  2.  {
  3.      hiveEP = new HiveEndPoint("thrift://" + HadoopProperties.HIVE_HOST + 
":" + HadoopProperties.HIVE_THRIFT_PORT, dbname, tablename, partitionVals);
  4.      try {
  5.          //generate timestamp agent string to be identifiable in logs
  6.          String agent = "tablename-";
  7.
  8.          Date curdate = Calendar.getInstance().getTime();
  9.          SimpleDateFormat format = new SimpleDateFormat("yyyyMMddHHmmss");
  10.         agent = agent + format.format(curdate);
  11.         hiveStream = hiveEP.newConnection(true, agent);
  12.     } catch 
(ConnectionError|InvalidPartition|InvalidTable|PartitionCreationFailed|ImpersonationFailed|InterruptedException|UndeclaredThrowableException
 e) {
  13.         log.error("HiveProcess constructor: " + e.getMessage());
  14.     }
  15. }

It successfully creates the connection so then it runs

  1.  public boolean processBatch(List<byte[]> toInsert) {
  2.      try {
  3.          StrictJsonWriter write = new StrictJsonWriter(hiveEP);
  4.          TransactionBatch txnBatch = 
hiveStream.fetchTransactionBatch(toInsert.size(), write);
  5.          //open the TransactionBatch
  6.          txnBatch.beginNextTransaction(); //Here is where the trouble 
starts
  7.          //loop through records for insertion
  8.          for(byte[] ins: toInsert) {
  9.              txnBatch.write(ins);
  10.         }
  11.         //commit batch
  12.         txnBatch.commit();
  13.         txnBatch.close();
  14.         //clean up connection, we are done until next time
  15.         hiveStream.close();
  16.         //if we made it to here, this is a success
  17.         return true;
  18.     } catch 
(StreamingException|InterruptedException|UndeclaredThrowableException e) {
  19.         log.error("HiveProcess processBatch " + e.getMessage());
  20.         //clean up connection, we are done until next time
  21.         hiveStream.close();
  22.         return false;
  23.     }
  24. }


  1.

When I switch to versions 2.0.0 of the dependencies I still get the warning but 
it does not raise a runtime error so everything can still go. So that would be 
fine and I would continue with the older version but since I have a newer 
version of Hive it wants agentInfo. I can't pass agentInfo to the older version 
of the streaming API. So I am in a quandry!

Can anyone help me figure out what is going wrong here and what I need to do to 
correct this?

Thanks!

Eva