[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446165#comment-16446165
]
Vineet Garg commented on HIVE-19157:
Pushed to branch-3. Hive jira for this commit is wrong (HIVE-19175). Will open
a jira to made an entry into errata.txt
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445847#comment-16445847
]
Ashutosh Chauhan commented on HIVE-19157:
-
That is commit : 1eea5a80ded2df33d57b2296b3bed98cb18383fd
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445345#comment-16445345
]
Vineet Garg commented on HIVE-19157:
[~ashutoshc] I am unable to find corresponding commit for this JIRA. Can you
point me to it?
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438142#comment-16438142
]
Ashutosh Chauhan commented on HIVE-19157:
-
+1
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438042#comment-16438042
]
slim bouguerra commented on HIVE-19157:
---
[~ashutoshc] Done
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.3.patch,
> HIVE-19157.4.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437949#comment-16437949
]
Ashutosh Chauhan commented on HIVE-19157:
-
yes.. it tried to run those tests but those timed out.. likely a temp glitch in
tests.
Can you create a RB for this?
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436456#comment-16436456
]
slim bouguerra commented on HIVE-19157:
---
[~ashutoshc] i see {code} TestMiniDruidCliDriver - did not produce a TEST-*.xml
file (likely timed out) (batchId=253){code} does that means the test did not
run?
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436396#comment-16436396
]
Hive QA commented on HIVE-19157:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918595/HIVE-19157.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 13274 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_u
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436300#comment-16436300
]
Hive QA commented on HIVE-19157:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} druid-handler: The patch generated 1 new + 138
unchanged - 1 fixed = 139 total (was 139) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10163/dev-support/hive-personality.sh
|
| git revision | master / b7c64b1 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10163/yetus/diff-checkstyle-druid-handler.txt
|
| modules | C: druid-handler U: druid-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10163/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434318#comment-16434318
]
slim bouguerra commented on HIVE-19157:
---
I have attache a second patch that has a different approach.
[^HIVE-19157.2.patch] In this version will check for the existence of metadata
directory first.
If such Directory is present on the file system then it must have data to load
thus the second list should return valid file names.
if it is absent then that means there is nothing to load.
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.2.patch, HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19157) Assert that Insert into Druid Table fails if the publishing of metadata by HS2 fails
[
https://issues.apache.org/jira/browse/HIVE-19157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433199#comment-16433199
]
slim bouguerra commented on HIVE-19157:
---
So as a fix am proposing to add a marker segment as part of the pre create
statement (probably also pre-insert) then make sure at least we find the marker
segment descriptor.
This seems to do the trick on my initial testing, looking for the community
feed back on this. Wondering if there is a better way to detect if 0 rows were
produced by the previous stages.
> Assert that Insert into Druid Table fails if the publishing of metadata by
> HS2 fails
>
>
> Key: HIVE-19157
> URL: https://issues.apache.org/jira/browse/HIVE-19157
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19157.patch
>
>
> The usual work flow of loading Data into Druid relies on the fact that HS2 is
> able to load Segments metadata from HDFS that are produced by LLAP/TEZ works.
> In some cases where HS2 is not able to perform `ls` on the HDFS path the
> insert into query will return success and will not insert any data.
> This bug was introduced at function {code}
> org.apache.hadoop.hive.druid.DruidStorageHandlerUtils#getCreatedSegments{code}
>
> when we added feature to allow create empty tables.
> {code}
> try {
> fss = fs.listStatus(taskDir);
> } catch (FileNotFoundException e) {
> // This is a CREATE TABLE statement or query executed for CTAS/INSERT
> // did not produce any result. We do not need to do anything, this is
> // expected behavior.
> return publishedSegmentsBuilder.build();
> }
> {code}
> Am still looking for the way to fix this, [~jcamachorodriguez]/[~ashutoshc]
> any idea what is the best way to detect that it is an empty create table
> statement?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
