[GitHub] [iceberg] NageshB82 commented on issue #2302: Reading from iceberg table through spark thrift server using jdbc taking larger time

[GitBox]

NageshB82 commented on issue #2302:
URL: https://github.com/apache/iceberg/issues/2302#issuecomment-803906213


   @RussellSpitzer  Here I am attaching the spark UI job and stages output when 
we are querying data from Account table from iceberg using simple jdbc program
   
   // get connection
           Connection con = 
DriverManager.getConnection("jdbc:hive2://<sparkthrift_server_ip>:10000/default",
 "", "");
           // create statement
           Statement stmt = con.createStatement();
           // execute statements
   ResultSet res = stmt.executeQuery("SELECT * FROM Account order by id limit 
20000;");
           
   Its almost taking minimum on 8 sec to fetch any 200 or 20000 records
   
   <img width="1527" alt="spark_job_read" 
src="https://user-images.githubusercontent.com/80257736/111967121-aac0b300-8b1d-11eb-9bea-8e3f28ef5c1a.png";>
   
   <img width="1533" alt="spark-stages-read" 
src="https://user-images.githubusercontent.com/80257736/111967159-b8763880-8b1d-11eb-9acc-0378f18c3e4d.png";>
   
   Here is Table creation statement we used for Account table in iceberg
   spark.sql(
           "CREATE TABLE if not exists Account(name  string,dn  string, 
accountCreationDate  string, disabled  long, forceChangePwd long, pwdLife long, 
numberLoginError long, lastChangePwd  string,  lastLogin  string, 
lastWrongLogin  string, state long, expire  string,  lastCertTime  string, 
creationDate  string, creationUser string, challengeCounter 
long,challengeFailedAttempt long, targetName string, applicationId string, 
lastAccessTime string, complianceStatus string, isDeleted  string, 
lastUpdatedTime  long, cumulativeScore string, currentScore string, critical 
string, high int, medium int, low int, firstOccurrence long, lastOccurrence 
long, id  string, sourceTypes  string, lastModifiedTime long, _id  string, type 
string, isimId string, externalId string) USING iceberg 
TBLPROPERTIES('engine.hive.enabled'='true')"
       )
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org