[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#15). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every,regardless of any one table failure. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp equaling the user-supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created the `-restore` table will not be removed. This should probably be done in a later commit. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 384 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/15 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 15 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#14). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every,regardless of any one table failure. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp equaling the user-supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created the `-restore` table will not be removed. This should probably be done in a later commit. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 384 insertions(+), 116 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/14 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 14 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#13). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every,regardless of any one table failure. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp equaling the user-supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created the `-restore` table will not be removed. This should probably be done in a later commit. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 343 insertions(+), 110 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/13 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 13 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#12). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every,regardless of any one table failure. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp equaling the user-supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created the `-restore` table will not be removed. This should probably be done in a later commit. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 342 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/12 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 12 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#11). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every,regardless of any one table failure. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp equaling the user-supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created the `-restore` table will not be removed. This should probably be done in a later commit. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 343 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/11 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 11 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#10). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created, it will not be removed. A list of failed/success full backups or restores will be printed at the end of each job. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 5 files changed, 344 insertions(+), 114 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/10 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 10 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Tony Foerster has restored this change. ( http://gerrit.cloudera.org:8080/10941 ) Change subject: Add error handling to Backup and Restore .. Restored Restoring after fixes -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 9 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Tony Foerster has abandoned this change. ( http://gerrit.cloudera.org:8080/10941 ) Change subject: Add error handling to Backup and Restore .. Abandoned I was considering this a sort of incremental change, but looking at it I think it needs more work. I'll open a new request after reworking. -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 9 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Tony Foerster has posted comments on this change. ( http://gerrit.cloudera.org:8080/10941 ) Change subject: Add error handling to Backup and Restore .. Patch Set 9: Build failure is not related to this changeset (OptionalSSL/TestRpc.TestClientConnectionMetrics/0) -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 9 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster Gerrit-Comment-Date: Mon, 23 Jul 2018 20:17:35 + Gerrit-HasComments: No
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#9). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created, it will not be removed. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 5 files changed, 346 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/9 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 9 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Tony Foerster has posted comments on this change. (
http://gerrit.cloudera.org:8080/10941 )
Change subject: Add error handling to Backup and Restore
..
Patch Set 8:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@35
PS6, Line 35: private val cause: Throwable
= null)
> nit: Just None.orNull is a bit unusual. Just null should work.
Done
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@46
PS6, Line 46: // TODO: Make parallel so each table isn't process serially?
> nit: s/vailid/valid
Done
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@57
PS6, Line 57: // Backup for this timestamp has already been
completed.
> I think we should only consider the case where getToMs == options.timestamp
Done
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@111
PS6, Line 111: val out = fs.create(hPath, /* overwrite= */ false)
> nit: extra spaces
Done
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala:
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@90
PS6, Line 90: val conf = session.sparkContext.hadoopConfiguration
> nit: extra spaces. I think a formatter is aligning the =. This should be fi
right it's my formatter...
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@103
PS6, Line 103:context: KuduContext):
Try[KuduTable] = Try {
> nit: explicit return type.
?? This has an explicit return type
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@115
PS6, Line 115: def main(args: Array[String]): Unit = {
> Where is this used? We definitely don't want the restore process dropping t
It was used when removing partial "${tableName}-restore" tables. I've removed
that part of the code for now but this was left behind. Fixed.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@136
PS6, Line 136:
> nit: s/note/not
Done
--
To view, visit http://gerrit.cloudera.org:8080/10941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691
Gerrit-Change-Number: 10941
Gerrit-PatchSet: 8
Gerrit-Owner: Tony Foerster
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tony Foerster
Gerrit-Comment-Date: Mon, 23 Jul 2018 19:36:21 +
Gerrit-HasComments: Yes
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#8). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created, it will not be removed. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- A java/kudu-backup/out/test/resources/log4j.properties M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 6 files changed, 369 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/8 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 8 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#7). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created, it will not be removed. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 5 files changed, 354 insertions(+), 153 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/7 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 7 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/10941 ) Change subject: Add error handling to Backup and Restore .. Patch Set 6: I think you are auto-reformatting all the files you edit. Using an auto-formatter for Scala is something I support, but it should be enabled and standardized as a separate patch. Until then please, disable it and keep the formatting changes to lines you are changing as a part of this feature. -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 6 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster Gerrit-Comment-Date: Fri, 20 Jul 2018 20:44:40 + Gerrit-HasComments: No
[kudu-CR] Add error handling to Backup and Restore
Grant Henke has posted comments on this change. (
http://gerrit.cloudera.org:8080/10941 )
Change subject: Add error handling to Backup and Restore
..
Patch Set 6:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@35
PS6, Line 35: private val cause: Throwable
= None.orNull)
nit: Just None.orNull is a bit unusual. Just null should work.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@46
PS6, Line 46: // TODO: Check if a vailid-looking result is in the directory
and don't retry
nit: s/vailid/valid
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@57
PS6, Line 57: if (metadata.getToMs >= options.timestampMs) {
I think we should only consider the case where getToMs == options.timestampMs
right now. In that case we can skip the full backup. Otherwise we should fail
saying a backup for a different time already exists (that should automatically
happen if you fall through here). Then the user can delete it or provide a new
path for the new backup.
Because we don't support any idea of incremental yet, that seams the most
straightforward to me.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@111
PS6, Line 111: val fs = hPath.getFileSystem(conf)
nit: extra spaces
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala:
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@90
PS6, Line 90: val conf = session.sparkContext.hadoopConfiguration
nit: extra spaces. I think a formatter is aligning the =. This should be fixed
in general.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@103
PS6, Line 103:context: KuduContext):
Try[KuduTable] = Try {
nit: explicit return type.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@115
PS6, Line 115: private def dropTable(tableName: String, kuduClient:
KuduClient): Try[Unit] =
Where is this used? We definitely don't want the restore process dropping
tables. I suppose we may want to drop a created table if it's restore failed
for some reason.
http://gerrit.cloudera.org:8080/#/c/10941/6/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@136
PS6, Line 136: s"${failures.length} were note restored, see previous
logs for details")
nit: s/note/not
--
To view, visit http://gerrit.cloudera.org:8080/10941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691
Gerrit-Change-Number: 10941
Gerrit-PatchSet: 6
Gerrit-Owner: Tony Foerster
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tony Foerster
Gerrit-Comment-Date: Fri, 20 Jul 2018 20:42:30 +
Gerrit-HasComments: Yes
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#6). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails the job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore Kudu table has already been created, it will not be removed. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 5 files changed, 368 insertions(+), 159 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/6 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 6 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
[kudu-CR] Add error handling to Backup and Restore
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10941 to look at the new patch set (#5). Change subject: Add error handling to Backup and Restore .. Add error handling to Backup and Restore A backup/restore will be attempted on every table. If any one table fails job will exit with a non-zero code. If a table is successfully backed up in the current job or the table metadata has a timestamp up to the current user supplied timestamp, the table will be marked successful. If a table fails to back up any partial data will be removed. If a table fails to restore and a restore table has already been created, it will not be removed. At the end of the job if any table has failed to backup/restore an exception is thrown and the job exits with a non-zero code. Spark SaveMode overwrite/append semantics are not handled as part of this commit. Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackupOptions.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 5 files changed, 368 insertions(+), 159 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/10941/5 -- To view, visit http://gerrit.cloudera.org:8080/10941 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I104fd604e12b70fcff9beace71ed4ef96b54d691 Gerrit-Change-Number: 10941 Gerrit-PatchSet: 5 Gerrit-Owner: Tony Foerster Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tony Foerster
