Hi Community, I need a quick consult for using the SET DDL while i'm using impala VIP for the load balancing.
Below are 2 ETL jobs, the first one is using hive and the second is impala and both have DDL operations. I will start with the 2nd as it should stright forward since the data inserted into the table_name2 by impala, so i think i don't need the invalidate statement but still need the DDL statement, is it right? should i add the set DDL after ALTER table statement? do you really need it as the insert isn't based on the ALTER table statement since it's dropping old partitions. For the 1st one where i'm using hive, I know i have to use invalidate metadata, i was thinking to use the set DDL=1 after each invalidate metadata statement, should i use after both statements, or it's enough to be used only in the last one? I read the documntation about this but wasn't sure if get this point right, so this is why i'm asking for consultation here, note a side i still think that i should use refresh statement after the insert while using invalidate after the ALTER statement. Thanks for your help in advance. RetentionDate=$(date --date="183 days ago" +%Y%m%d) impala-shell -i "$LPIMPALA" -q "ALTER TABLE table_name DROP IF EXISTS PARTITION (concat(year,month,day)<'$RetentionDate');" impala-shell -i "$LPIMPALA" -q "INVALIDATE METADATA table_name;" hive -e" set parquet.compression=SNAPPY; set hive.merge.mapredfiles=true; set hive.mapred.mode=nonstrict; set hive.exec.dynamic.partition.mode=nonstrict; INSERT OVERWRITE TABLE table_name PARTITION (year, month, day) .... .... .... .... ;" impala-shell -i "$LPIMPALA" -q "INVALIDATE METADATA table_name;" impala-shell -i "$LPIMPALA" -q "compute INCREMENTAL STATS table_name;" ======================================================= 2nd one. impala-shell -i "$LPIMPALA" -q "ALTER TABLE table_name2 DROP IF EXISTS PARTITION (dt<'$RetentionDate');" impala-shell -i "$LPIMPALA" -q "INVALIDATE METADATA table_name2;" impala-shell -B -i "$LPIMPALA" -q" INSERT OVERWRITE TABLE table_name2 PARTITION (dt) .... .... .... .... " impala-shell -i "$LPIMPALA" -q "INVALIDATE METADATA table_name2;" -- Take Care Fawze Abujaber
