[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77575744
--- Diff: tools/bin/hawqregister ---
@@ -336,8 +338,25 @@ def insert_metadata_into_database(dburl, databasename,
tablename, seg_name, firs
conn.commit()
conn.close()
except DatabaseError, ex:
-logger.error('Failed to connect to database, this script can only
be run when the database is up')
-move_files_in_hdfs(database, tablename, files, firstsegno,
tabledir, False)
+logger.error('Failed to execute query "%s"' % query)
+move_files_in_hdfs(databasename, tablename, files, firstsegno,
tabledir, False)
+sys.exit(1)
+
+def update_metadata_into_database(dburl, seg_name, files, eofs):
+'''Update the catalog table in --force case'''
+try:
+query = "set allow_system_table_mods='dml';"
+query += "begin transaction;"
+segno_lst = [f.split('/')[-1] for f in files]
+for i, eof in enumerate(eofs):
+query += "update pg_aoseg.%s set eof = '%s' where segno =
'%s';" % (seg_name, eof, segno_lst[i])
--- End diff --
To unify all use cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #874: Remove Thrift 0.9.3 from source tree - HAWQ-959
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/874 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #874: Remove Thrift 0.9.3 from source tree - HAWQ-959
Github user xunzhang commented on the issue: https://github.com/apache/incubator-hawq/pull/874 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (HAWQ-1032) Bucket number of newly added partition is not consistent with parent table.
[
https://issues.apache.org/jira/browse/HAWQ-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lili Ma updated HAWQ-1032:
--
Description:
Failure Case
{code}
set deafult_hash_table_bucket_number = 12;
CREATE TABLE sales3 (id int, date date, amt decimal(10,2)) DISTRIBUTED
BY (id) PARTITION BY
RANGE (date) ( START (date
'2008-01-01') INCLUSIVEEND (date
'2009-01-01') EXCLUSIVE EVERY
(INTERVAL '1 day') );
set default_hash_table_bucket_number = 16;
ALTER TABLE sales3 ADD PARTITION START (date
'2009-03-01') INCLUSIVE END (date
'2009-04-01') EXCLUSIVE;
{code}
The newly added partition with buckcet number 16 is not consistent with parent
partition.
was:
Failure Case
{code}
set deafult_hash_table_bucket_number = 12;
CREATE TABLE sales3 (id int, date date, amt decimal(10,2)) DISTRIBUTED
BY (id) PARTITION BY
RANGE (date) ( START (date
'2008-01-01') INCLUSIVEEND (date
'2009-01-01') EXCLUSIVE EVERY
(INTERVAL '1 day') );
set deafult_hash_table_bucket_number = 16;
ALTER TABLE sales3 ADD PARTITION START (date
'2009-03-01') INCLUSIVE END (date
'2009-04-01') EXCLUSIVE;
{code}
The newly added partition with buckcet number 16 is not consistent with parent
partition.
> Bucket number of newly added partition is not consistent with parent table.
> ---
>
> Key: HAWQ-1032
> URL: https://issues.apache.org/jira/browse/HAWQ-1032
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
> Fix For: 2.0.1.0-incubating
>
>
> Failure Case
> {code}
> set deafult_hash_table_bucket_number = 12;
> CREATE TABLE sales3 (id int, date date, amt decimal(10,2))
> DISTRIBUTED BY (id)
> PARTITION BY RANGE (date)
> ( START (date '2008-01-01') INCLUSIVE
>END (date '2009-01-01') EXCLUSIVE
>EVERY (INTERVAL '1 day') );
> set default_hash_table_bucket_number = 16;
> ALTER TABLE sales3 ADD PARTITION START
> (date '2009-03-01') INCLUSIVE END
> (date '2009-04-01') EXCLUSIVE;
> {code}
> The newly added partition with buckcet number 16 is not consistent with
> parent partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-hawq pull request #888: HAWQ-904. Fix hawq config help messages.
Github user asfgit closed the pull request at: https://github.com/apache/incubator-hawq/pull/888 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #888: HAWQ-904. Fix hawq config help messages.
Github user yaoj2 commented on the issue: https://github.com/apache/incubator-hawq/pull/888 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #888: HAWQ-904. Fix hawq config help messages.
Github user radarwave commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/888#discussion_r77566477 --- Diff: tools/bin/hawq --- @@ -143,7 +143,7 @@ def main(): cmd = "%s; hawq_ctl %s %s" % (source_hawq_env, hawq_command, sub_args) result = local_run(cmd) elif hawq_command == "config": -if second_arg in ['', 'help', '--help']: +if second_arg in ['', 'help', '--help', '-h']: --- End diff -- Thanks, fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564199
--- Diff: tools/bin/hawqregister ---
@@ -336,8 +338,25 @@ def insert_metadata_into_database(dburl, databasename,
tablename, seg_name, firs
conn.commit()
conn.close()
except DatabaseError, ex:
-logger.error('Failed to connect to database, this script can only
be run when the database is up')
-move_files_in_hdfs(database, tablename, files, firstsegno,
tabledir, False)
+logger.error('Failed to execute query "%s"' % query)
+move_files_in_hdfs(databasename, tablename, files, firstsegno,
tabledir, False)
+sys.exit(1)
+
+def update_metadata_into_database(dburl, seg_name, files, eofs):
+'''Update the catalog table in --force case'''
+try:
+query = "set allow_system_table_mods='dml';"
+query += "begin transaction;"
+segno_lst = [f.split('/')[-1] for f in files]
+for i, eof in enumerate(eofs):
+query += "update pg_aoseg.%s set eof = '%s' where segno =
'%s';" % (seg_name, eof, segno_lst[i])
--- End diff --
Why use "update" instead of "delete" and re"insert"?? What if the new files
are more than existing files??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564087
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with
(appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2),
(3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string)
HAWQ_DB + " -o tmp.yml
testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string)
HAWQ_DB + " -c tmp.yml
testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 3);
+
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with
(appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(4), (5),
(6), (7);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string)
HAWQ_DB + " -o tmp.yml
testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string)
HAWQ_DB + " -c tmp.yml --force
testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 4);
--- End diff --
Should here throw error, indicating the existing files for table
'new_simple_register_table_force' are not included in .yml file???
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564020
--- Diff: tools/bin/hawqregister ---
@@ -388,11 +415,31 @@ if __name__ == '__main__':
logger.error('Files to be registered in this case should not
be the same with table path.')
sys.exit(1)
+do_not_move, files_update, sizes_update = False, [], []
+if force_mode:
+existed_files = get_files_in_hdfs(filepath)
+if len(files) == len(exited_files):
--- End diff --
Is this a spelling error? Should be existed_files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #884: HAWQ-1032. Set bucket number of child part...
Github user zhangh43 closed the pull request at: https://github.com/apache/incubator-hawq/pull/884 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #888: HAWQ-904. Fix hawq config help messages.
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/888#discussion_r77489007 --- Diff: tools/bin/hawq --- @@ -143,7 +143,7 @@ def main(): cmd = "%s; hawq_ctl %s %s" % (source_hawq_env, hawq_command, sub_args) result = local_run(cmd) elif hawq_command == "config": -if second_arg in ['', 'help', '--help']: +if second_arg in ['', 'help', '--help', '-h']: --- End diff -- Also fix for "hawq activate"? elif hawq_command == "activate": if second_arg in ['', 'help', '--help']: print ACTIVE_HELP sys.exit(1) cmd = "%s; hawq_ctl %s %s" % (source_hawq_env, hawq_command, sub_args) result = local_run(cmd) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #888: HAWQ-904. Fix hawq config help messages.
GitHub user radarwave opened a pull request: https://github.com/apache/incubator-hawq/pull/888 HAWQ-904. Fix hawq config help messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/radarwave/incubator-hawq confighelp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/888.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #888 commit cd2423947f74c1ab11f14e6b277870b22152b4d8 Author: rlei Date: 2016-09-05T08:44:01Z HAWQ-904. Fix hawq config help messages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #887: HAWQ-1032. Fix bug of wrong parameter in G...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-hawq/pull/887 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #887: HAWQ-1032. Fix bug of wrong parameter in GpPolicy...
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/887 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #887: HAWQ-1032. Fix bug of wrong parameter in GpPolicy...
Github user liming01 commented on the issue: https://github.com/apache/incubator-hawq/pull/887 +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #887: HAWQ-1032. Fix bug of wrong parameter in G...
GitHub user zhangh43 opened a pull request: https://github.com/apache/incubator-hawq/pull/887 HAWQ-1032. Fix bug of wrong parameter in GpPolicyFetch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhangh43/incubator-hawq hawq1032bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/887.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #887 commit 268b4e5e0c176fdb5d51d36cd876bde4a8b3c598 Author: hzhang2 Date: 2016-09-05T07:03:28Z HAWQ-1032. Fix bug of wrong parameter in GpPolicyFetch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
