Hello Mike Percy, Kudu Jenkins, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13430
to look at the new patch set (#2).
Change subject: [backup] KUDU-2786 Parallelize tables for backup and restore
......................................................................
[backup] KUDU-2786 Parallelize tables for backup and restore
This patch adds a hidden, experimental option to run backups and
restores in parallel across tables. Managing resources across parallel
backups and restores is very difficult: the sizes of tables in terms of
number of tablets and size of tables can vary by orders of magnitude
across a cluster, and there are many resources which may be constrained
depending on many factors: CPU, memory, disk I/O, network, number of
executors available. This patch doesn't do resource management. It will
kick off the jobs in parallel, and it's up to Spark to manage the
resources of parallel jobs. Maybe this will work well, maybe it won't...
that's why this is just experimental.
I tested manually on a Spark cluster to verify that jobs are actually
run in parallel.
Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
4 files changed, 69 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13430/2
--
To view, visit http://gerrit.cloudera.org:8080/13430
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e
Gerrit-Change-Number: 13430
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wdberke...@gmail.com>
Gerrit-Reviewer: Grant Henke <granthe...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mpe...@apache.org>
