[jira] [Commented] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624133#comment-15624133 ] Lili Ma commented on HAWQ-1034: --- Repair mode can be thought of particular case of force mode. 1) Force mode registers the files according to yaml configuration file, erase all the records in catalog (pg_aoseg.pg_aoseg(paqseg)_$relid) and re-implement catalog insert. It requires HDFS files for the table be included in yaml configuation file. 2) Repair mode also registers files according to yaml configuration file, erase the catalog records and re-insert. But it doesn't require all the HDFS files for the table be included in yaml configuration file. It will directly delete those files which are under the table directory but not included in yaml configuration file. I'm a little concerned about directly deleting HDFS files, say, if user uses repair mode by mistake, his/her data may be deleted. So, what if we just allow them to use force mode, and throw error for files under the directory but not included in yaml configuration file. If user does think the files are unnecessary, he/she can delete the files by himself/herself. The workaround for supporting repair mode use --force option: 1) If there is no added files since last checkpoint where the yaml configuration file is generated, force mode can directly handle it. 2) If there are some added files since last checkpoint which the user does want to delete, we can output those file information in force mode so that users can delete those files by themselves and then do register force mode again. Since we can use force mode to implement repair feature, we will remove existing code for repair mode and close this JIRA. Thanks > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621371#comment-15621371 ] Chunling Wang commented on HAWQ-1034: - The reason why we remove these code is that removing data in table directory in repair mode will cause risk to lose data. So we decided to not remove data in table directory in repair mode, which can be replaced by force mode. > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621348#comment-15621348 ] Chunling Wang commented on HAWQ-1034: - The code and test cases for repair mode have been removed by GitHub Pull Request #986. > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
