[jira] [Commented] (TAJO-1346) Create dynamic partitions to CatalogStore by running insert query or CTAS query.

[ASF GitHub Bot (JIRA)]

    [ 
https://issues.apache.org/jira/browse/TAJO-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642205#comment-14642205
 ] 

ASF GitHub Bot commented on TAJO-1346:
--------------------------------------

Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/630#discussion_r35503742
  
    --- Diff: 
tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
 ---
    @@ -1245,116 +1242,118 @@ private void addNewColumn(int tableId, 
CatalogProtos.ColumnProto columnProto) th
     
       public void addPartition(int tableId, CatalogProtos.PartitionDescProto 
partition) throws CatalogException {
         Connection conn = null;
    -    PreparedStatement pstmt = null;
    -    final String ADD_PARTITION_SQL =
    -      "INSERT INTO " + TB_PARTTIONS
    -        + " (" + COL_TABLES_PK + ", PARTITION_NAME, PATH) VALUES (?,?,?)";
    -
    -    final String ADD_PARTITION_KEYS_SQL =
    -      "INSERT INTO " + TB_PARTTION_KEYS + " (" + COL_PARTITIONS_PK + ", " 
+ COL_COLUMN_NAME + ", "
    -      + COL_PARTITION_VALUE + ") VALUES (?,?,?)";
    +    Statement stmt = null;
     
         try {
    -
    -      if (LOG.isDebugEnabled()) {
    -        LOG.debug(ADD_PARTITION_SQL);
    -      }
    -
           conn = getConnection();
    -      pstmt = conn.prepareStatement(ADD_PARTITION_SQL);
    -      pstmt.setInt(1, tableId);
    -      pstmt.setString(2, partition.getPartitionName());
    -      pstmt.setString(3, partition.getPath());
    -      pstmt.executeUpdate();
    -      pstmt.close();
    +      conn.setAutoCommit(false);
    +      stmt = conn.createStatement();
     
    -      if (partition.getPartitionKeysCount() > 0) {
    -        pstmt = conn.prepareStatement(ADD_PARTITION_KEYS_SQL);
    -        int partitionId = getPartitionId(tableId, 
partition.getPartitionName());
    -        addPartitionKeys(pstmt, partitionId, partition);
    -        pstmt.executeBatch();
    +      addPartition(tableId, partition, stmt, new StringBuilder());
    +      addPartitionKeys(tableId, partition, stmt, new StringBuilder());
    +      stmt.executeBatch();
    +
    +      if (conn != null) {
    +        conn.commit();
           }
         } catch (SQLException se) {
    +      if (conn != null) {
    +        try {
    +          conn.rollback();
    +        } catch (SQLException e) {
    +          LOG.error(e, e);
    +        }
    +      }
           throw new TajoInternalError(se);
         } finally {
    -      CatalogUtil.closeQuietly(pstmt);
    +      CatalogUtil.closeQuietly(stmt);
         }
       }
     
    -  public int getPartitionId(int tableId, String partitionName) throws 
CatalogException {
    -    Connection conn = null;
    -    ResultSet res = null;
    -    PreparedStatement pstmt = null;
    -    int retValue = -1;
    -
    +  public void addPartition(int tableId, CatalogProtos.PartitionDescProto 
partition,
    +                           Statement stmt, StringBuilder sb) throws 
CatalogException {
         try {
    -      String sql = "SELECT " + COL_PARTITIONS_PK + " FROM " + TB_PARTTIONS 
+
    -        " WHERE " + COL_TABLES_PK + " = ? AND PARTITION_NAME = ? ";
    +      sb.delete(0, sb.length());
     
    -      if (LOG.isDebugEnabled()) {
    -        LOG.debug(sql);
    -      }
    -
    -      conn = getConnection();
    -      pstmt = conn.prepareStatement(sql);
    -      pstmt.setInt(1, tableId);
    -      pstmt.setString(2, partitionName);
    -      res = pstmt.executeQuery();
    +      sb.append("INSERT INTO ").append(TB_PARTTIONS).append(" ");
    +      sb.append(" (").append(COL_TABLES_PK).append(", PARTITION_NAME, 
PATH)");
    +      sb.append(" VALUES (").append(tableId).append(", 
'").append(partition.getPartitionName());
    +      sb.append("' , '").append(partition.getPath()).append("')");
     
    -      if (res.next()) {
    -        retValue = res.getInt(1);
    -      }
    +      stmt.addBatch(sb.toString());
         } catch (SQLException se) {
           throw new TajoInternalError(se);
    -    } finally {
    -      CatalogUtil.closeQuietly(pstmt, res);
         }
    -    return retValue;
       }
     
    -  private void addPartitionKeys(PreparedStatement pstmt, int partitionId, 
PartitionDescProto partition) throws
    -    SQLException {
    -    for (int i = 0; i < partition.getPartitionKeysCount(); i++) {
    -      PartitionKeyProto partitionKey = partition.getPartitionKeys(i);
    +  public void addPartitionKeys(int tableId, 
CatalogProtos.PartitionDescProto partition,
    --- End diff --
    
    StringBuilder parameter seems to be used to reuse StringBuilder object. I'd 
like to recommend that do not try code-level optimization until the code part 
actually causes bottleneck. This kind of optimization is not effective whereas 
it is harmful for code readability.


> Create dynamic partitions to CatalogStore by running insert query or CTAS 
> query.
> --------------------------------------------------------------------------------
>
>                 Key: TAJO-1346
>                 URL: https://issues.apache.org/jira/browse/TAJO-1346
>             Project: Tajo
>          Issue Type: Sub-task
>          Components: Physical Operator
>            Reporter: Jaehwa Jung
>            Assignee: Jaehwa Jung
>
> Currently, If new partitions are added to HDFS by ColPartitionStoreExec, 
> CatalogStore will not be aware of these partitions. So, Tajo need to load 
> newly added partitions automatically to CatalogStore by running insert query 
> or CTAS query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)