[kudu-CR] KUDU-2542: add initial authorization token impl
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11750 ) Change subject: KUDU-2542: add initial authorization token impl .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11750/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11750/3//COMMIT_MSG@19 PS3, Line 19: : The tokens leverage the same token signer as the authentication tokens, : though with the token validity interval configured via the new flag : --authz_token_validity_seconds. > I'm curious why you chose to use a new gflag rather than generalizing the e They serve to answer fundamentally different questions: how frequently will I refresh my trust that you are who you say you are, and how frequently will I refresh my trust that you are allowed to perform actions XYZ? The way we expect to refresh the trust for each of these questions is different (e.g. via KDC or via Ranger/Sentry), so it doesn't seem unreasonable to vary the intervals as well. -- To view, visit http://gerrit.cloudera.org:8080/11750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id28747ec38675abdf50dce1e7c176d29213e370f Gerrit-Change-Number: 11750 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Sat, 27 Oct 2018 03:45:05 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Hello Will Berkeley, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, Greg Solovyev, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11788 to look at the new patch set (#8). Change subject: [examples] Add basic Spark example written in Scala .. [examples] Add basic Spark example written in Scala This patch adds a basic Kudu client that utilizes both Kudu Java APIs, as well as Spark SQL APIs. It will allow customers to pull down the pom.xml and scala source, then build and execute from their local machine. Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f --- A examples/scala/spark-example/README.adoc A examples/scala/spark-example/pom.xml A examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala 3 files changed, 292 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/11788/8 -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 8 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 7: So the snag I hit previously ended up being KUDU-2259, which caused the job to fail since the token was reissued by the master and it attempted to use a different name (despite there being no auth configured..) It looks like this was supposed to be fixed in 1.7, but I had to update the pom to pull 1.7.1 or 1.8.0 to get it working. I don't know how changes in Kudu directly impact the kudu-spark lib, but it's resolved nonetheless. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 7 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Sat, 27 Oct 2018 02:04:47 + Gerrit-HasComments: No
[kudu-CR] [examples] Add basic Spark example written in Scala
Hello Will Berkeley, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, Greg Solovyev, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11788 to look at the new patch set (#7). Change subject: [examples] Add basic Spark example written in Scala .. [examples] Add basic Spark example written in Scala This patch adds a basic Kudu client that utilizes both Kudu Java APIs, as well as Spark SQL APIs. It will allow customers to pull down the pom.xml and scala source, then build and execute from their local machine. Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f --- A examples/scala/spark-example/README.adoc A examples/scala/spark-example/pom.xml A examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala 3 files changed, 287 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/11788/7 -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 7 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 6: Pushed the additional instructions for running this as a spark2-submit job, however there was a small snag I encountered that I'm still working through. The job will actually run successfully and print out as expected, but will fail when closing down due to how I'm defining the SparkSession's 'master' instance. I'm working through how to mitigate that, and still have it be executable as both a java standalone app and via spark2-submit. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 6 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Sat, 27 Oct 2018 01:17:42 + Gerrit-HasComments: No
[kudu-CR] [examples] Add basic Spark example written in Scala
Hello Will Berkeley, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, Greg Solovyev, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11788 to look at the new patch set (#6). Change subject: [examples] Add basic Spark example written in Scala .. [examples] Add basic Spark example written in Scala This patch adds a basic Kudu client that utilizes both Kudu Java APIs, as well as Spark SQL APIs. It will allow customers to pull down the pom.xml and scala source, then build and execute from their local machine. Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f --- A examples/scala/spark-example/README.adoc A examples/scala/spark-example/pom.xml A examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala 3 files changed, 287 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/11788/6 -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 6 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley
[kudu-CR] [examples] Add basic Spark example written in Scala
Hello Will Berkeley, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, Greg Solovyev, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11788 to look at the new patch set (#5). Change subject: [examples] Add basic Spark example written in Scala .. [examples] Add basic Spark example written in Scala This patch adds a basic Kudu client that utilizes both Kudu Java APIs, as well as Spark SQL APIs. It will allow customers to pull down the pom.xml and scala source, then build and execute from their local machine. Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f --- A examples/scala/spark-example/README.adoc A examples/scala/spark-example/pom.xml A examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala 3 files changed, 287 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/88/11788/5 -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 5 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley
[kudu-CR] KUDU-2542: add initial authorization token impl
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/11750 ) Change subject: KUDU-2542: add initial authorization token impl .. Patch Set 3: Code-Review+1 LGTM, but it would be nice if Alexey takes another look. -- To view, visit http://gerrit.cloudera.org:8080/11750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id28747ec38675abdf50dce1e7c176d29213e370f Gerrit-Change-Number: 11750 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 23:52:16 + Gerrit-HasComments: No
[kudu-CR] [location awareness] Add location info in ksck report
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/11422 ) Change subject: [location_awareness] Add location info in ksck report .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/11422/11/src/kudu/tools/ksck.h File src/kudu/tools/ksck.h: http://gerrit.cloudera.org:8080/#/c/11422/11/src/kudu/tools/ksck.h@388 PS11, Line 388: std::string location_ > The way I tested in ksck-test does make modification to the location_ (ksck Ah, I missed that piece where location is updated in the test. Sure, in that case it does not make sense to add 'const' there. -- To view, visit http://gerrit.cloudera.org:8080/11422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ideff2dd4975c99a1135002624de2620fb95c Gerrit-Change-Number: 11422 Gerrit-PatchSet: 11 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 23:10:33 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > I guess, my point is that it would be nice if this example was applicable t I've got this running with the System.getProperty() call on another spark2 cluster. I'll write up some instructions and include them here so that customers don't have to do this same song and dance as we are :) -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 22:22:34 + Gerrit-HasComments: Yes
[kudu-CR] [location awareness] Add location info in ksck report
Fengling Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11422 ) Change subject: [location_awareness] Add location info in ksck report .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/11422/11/src/kudu/tools/ksck.h File src/kudu/tools/ksck.h: http://gerrit.cloudera.org:8080/#/c/11422/11/src/kudu/tools/ksck.h@388 PS11, Line 388: std::string location_ > nit: it seems this field is set only in the constructor and is not changing The way I tested in ksck-test does make modification to the location_ (ksck-test.cc, line 1605). Would you suggest me to keep it or write another function that constructs a tablet server with location in order to test? -- To view, visit http://gerrit.cloudera.org:8080/11422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ideff2dd4975c99a1135002624de2620fb95c Gerrit-Change-Number: 11422 Gerrit-PatchSet: 11 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 22:09:56 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > I tried running this on Spark2 on Yarn after replacing how parameters are p I guess, my point is that it would be nice if this example was applicable to CDH. Otherwise, it should probably just use local execution. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 22:03:04 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > I ran the job locally - worked fine. Then, I tried running it against a Spa I tried running this on Spark2 on Yarn after replacing how parameters are passed (args instead of System.getProperty) and it almost worked. It broke with "java.lang.NoClassDefFoundError: org/apache/commons/dbcp/ConnectionFactory" -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 22:00:06 + Gerrit-HasComments: Yes
[kudu-CR](gh-pages) Fix 1.8.0 download page
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11803 ) Change subject: Fix 1.8.0 download page .. Fix 1.8.0 download page It seems manual mirror choosing page is necessary instead of the automatic download action. [1] https://lists.apache.org/thread.html/cfdfae1286d919963e573640f3305b05acbe4ae49e1754fd538ae1c1@%3Cdev.kudu.apache.org%3E Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 Reviewed-on: http://gerrit.cloudera.org:8080/11803 Reviewed-by: Grant Henke Tested-by: Attila Bukor --- M releases/1.8.0/index.md 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Grant Henke: Looks good to me, approved Attila Bukor: Verified -- To view, visit http://gerrit.cloudera.org:8080/11803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: merged Gerrit-Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 Gerrit-Change-Number: 11803 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke
[kudu-CR](gh-pages) Fix 1.8.0 download page
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/11803 ) Change subject: Fix 1.8.0 download page .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 Gerrit-Change-Number: 11803 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Comment-Date: Fri, 26 Oct 2018 21:56:54 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Fix 1.8.0 download page
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11803 ) Change subject: Fix 1.8.0 download page .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 Gerrit-Change-Number: 11803 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Comment-Date: Fri, 26 Oct 2018 21:54:52 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Fix 1.8.0 download page
Hello Grant Henke, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/11803 to review the following change. Change subject: Fix 1.8.0 download page .. Fix 1.8.0 download page It seems manual mirror choosing page is necessary instead of the automatic download action. [1] https://lists.apache.org/thread.html/cfdfae1286d919963e573640f3305b05acbe4ae49e1754fd538ae1c1@%3Cdev.kudu.apache.org%3E Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 --- M releases/1.8.0/index.md 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/11803/1 -- To view, visit http://gerrit.cloudera.org:8080/11803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: newchange Gerrit-Change-Id: I76453996cc9ddb97c6753477780f73796298c1c1 Gerrit-Change-Number: 11803 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Grant Henke
[kudu-CR](gh-pages) Fix 1.8.0 download page
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/11802 ) Change subject: Fix 1.8.0 download page .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Gerrit-Change-Number: 11802 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Fri, 26 Oct 2018 21:20:33 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Fix 1.8.0 download page
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11802 ) Change subject: Fix 1.8.0 download page .. Fix 1.8.0 download page The release announcement of Kudu 1.8.0 was rejected on announce@a.o due to not conforming to requirements[1] This commit addresses all of these concerns. [1] https://lists.apache.org/thread.html/9cf2fcf4e919520919f1fc860231d244781cf63ee40067ed6f47494c@%3Cdev.kudu.apache.org%3E Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Reviewed-on: http://gerrit.cloudera.org:8080/11802 Reviewed-by: Grant Henke Tested-by: Attila Bukor --- M releases/1.8.0/index.md 1 file changed, 12 insertions(+), 3 deletions(-) Approvals: Grant Henke: Looks good to me, approved Attila Bukor: Verified -- To view, visit http://gerrit.cloudera.org:8080/11802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: merged Gerrit-Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Gerrit-Change-Number: 11802 Gerrit-PatchSet: 3 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy
[kudu-CR](gh-pages) Fix 1.8.0 download page
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11802 ) Change subject: Fix 1.8.0 download page .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Gerrit-Change-Number: 11802 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Fri, 26 Oct 2018 21:19:19 + Gerrit-HasComments: No
[kudu-CR] docs: Add scripts to automate web site update for releases
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/8800 ) Change subject: docs: Add scripts to automate web site update for releases .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template File docs/support/scripts/release_index.md.template: http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@35 PS1, Line 35: http://www.apache.org/closer.cgi?filename=kudu/%VERSION%/apache-kudu-%VERSION%.tar.gz=download > link should be https://archive.apache.org/dist/kudu/%VERSION%/apache-kudu-% never mind, it seems we need to use this chooser script. It could still be https though http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@36 PS1, Line 36: www > same here, link should point to archive.apache.org instead of www.apache.or never mind, it seems we need to use www.a.o http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@40 PS1, Line 40: You can use the [KEYS file](https://www.apache.org/dist/kudu/KEYS) to verify the included GPG signature. we need to add more detailed instructions on how to verify the integrity: https://lists.apache.org/thread.html/9cf2fcf4e919520919f1fc860231d244781cf63ee40067ed6f47494c@%3Cdev.kudu.apache.org%3E example: https://gerrit.cloudera.org/c/11802/ -- To view, visit http://gerrit.cloudera.org:8080/8800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If55437e0561f0bf8e5e0ec0ae6e5bfca0311fb7d Gerrit-Change-Number: 8800 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Fri, 26 Oct 2018 21:11:33 + Gerrit-HasComments: Yes
[kudu-CR](gh-pages) Fix 1.8.0 download page
Hello Mike Percy, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11802 to look at the new patch set (#2). Change subject: Fix 1.8.0 download page .. Fix 1.8.0 download page The release announcement of Kudu 1.8.0 was rejected on announce@a.o due to not conforming to requirements[1] This commit addresses all of these concerns. [1] https://lists.apache.org/thread.html/9cf2fcf4e919520919f1fc860231d244781cf63ee40067ed6f47494c@%3Cdev.kudu.apache.org%3E Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 --- M releases/1.8.0/index.md 1 file changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/02/11802/2 -- To view, visit http://gerrit.cloudera.org:8080/11802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Gerrit-Change-Number: 11802 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy
[kudu-CR](gh-pages) Fix 1.8.0 download page
Hello Mike Percy, Grant Henke, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/11802 to review the following change. Change subject: Fix 1.8.0 download page .. Fix 1.8.0 download page The release announcement of Kudu 1.8.0 was rejected on announce@a.o due to not conforming to requirements[1] This commit addresses all of these concerns. [1] https://lists.apache.org/thread.html/9cf2fcf4e919520919f1fc860231d244781cf63ee40067ed6f47494c@%3Cdev.kudu.apache.org%3E Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 --- M releases/1.8.0/index.md 1 file changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/02/11802/1 -- To view, visit http://gerrit.cloudera.org:8080/11802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: newchange Gerrit-Change-Id: I0ef0c7be1f07e8c5c290fb6b975bb8c46a2ec864 Gerrit-Change-Number: 11802 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Improve consensus queue overflow logging Suppose tablet server X is a leader of T tablets for which tablet server Y is a follower. The relevant situation is when T is on the order of 100-1000. If Y strains under its consensus load and falls behind processing consensus service requests, UpdateConsensus requests from the leader will get rejected and cause a message to be logged on the leader X for each of the T tablets. The message looks like: W1022 17:20:59.767554 13057 consensus_peers.cc:422] T 9255fdf03ad4451e9fcd62f26741bfe6 P 892cc0d4442c4cdaaf633ed2732f9246 -> Peer dc0af5867d52468f8fd47abf13c08040 (tablet_server_Y.kudu.com:7050): Couldn't send request to peer dc0af5867d52468f8fd47abf13c08040 for tablet 9255fdf03ad4451e9fcd62f26741bfe6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 10.1.1.1:55528 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1 times. Y's consensus service pool also logs the same thing, but it doesn't have the information about the tablet id or peer ids available to it, and it is throttled to occur no more than once per second: W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. This patch attempts to reduce the spam of the first message in the logs by throttling it to occur once every 5 retries. It still is logged for every tablet peer, but those messages are useful if one wants to trace the history of a particular tablet. I also added the throttling messages to Y's output, so it's now W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 5 similar messages] when e.g. 5 other messages have been suppressed. Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Reviewed-on: http://gerrit.cloudera.org:8080/11801 Tested-by: Will Berkeley Reviewed-by: Alexey Serbin --- M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_peers.h M src/kudu/rpc/service_pool.cc M src/kudu/util/logging.h 4 files changed, 22 insertions(+), 10 deletions(-) Approvals: Will Berkeley: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Will Berkeley
[kudu-CR] [examples] Add basic Spark example written in Scala
Grant Henke has posted comments on this change. (
http://gerrit.cloudera.org:8080/11788 )
Change subject: [examples] Add basic Spark example written in Scala
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc
File examples/scala/spark-example/README.adoc:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41
PS4, Line 41: SparkMaster
> I ran the job locally - worked fine. Then, I tried running it against a Spa
We don't support Spark 1 anymore. That was dropped a few versions back.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala
File
examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@71
PS4, Line 71: val upsertUsers = Array(User("newUserA", 1234),
User("userC", ))
> Yep, I am repeating a value from above (the id=1234). This will upsert 1234
oh, my bad I was looking at the names.
--
To view, visit http://gerrit.cloudera.org:8080/11788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f
Gerrit-Change-Number: 11788
Gerrit-PatchSet: 4
Gerrit-Owner: Mitch Barnett
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Attila Bukor
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Greg Solovyev
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Fri, 26 Oct 2018 20:34:29 +
Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc File src/kudu/consensus/consensus_peers.cc: http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc@458 PS1, Line 458: . T > See L446- it's either an empty string or an actual messages depending on th Ah, indeed -- I missed that part. -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 20:29:15 + Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Removed reviewer Kudu Jenkins with the following votes: * Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Will Berkeley
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 2: Verified+1 Failure is (unrealted) KUDU-1736. -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 20:19:27 + Gerrit-HasComments: No
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > This should work without issue in a spark-submit job, I tested that previou I ran the job locally - worked fine. Then, I tried running it against a Spark 1.6 standalone cluster (CHD5.15.2) and got an error "java.io.StreamCorruptedException: invalid stream header: 01000C31" - because of the mismatch between Spark 2 and Spark 1.6. The way I submitted SparkMaster URL is exactly how your colleague described: java -DKuduMasters=greg-kudu-5152-1.vpc.cloudera.com:7051 -DSparkMaster=spark://greg-kudu-5152-1.vpc.cloudera.com:7077 -jar target/kudu-spark-example-1.0-SNAPSHOT.jar I didn't try running it against Spark2 on yarn yet, that will require rewriting the example code a bit. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 20:18:00 + Gerrit-HasComments: Yes
[kudu-CR] docs: Add scripts to automate web site update for releases
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/8800 ) Change subject: docs: Add scripts to automate web site update for releases .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template File docs/support/scripts/release_index.md.template: http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@35 PS1, Line 35: http://www.apache.org/closer.cgi?filename=kudu/%VERSION%/apache-kudu-%VERSION%.tar.gz=download link should be https://archive.apache.org/dist/kudu/%VERSION%/apache-kudu-%VERSION%.tar.gz http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@36 PS1, Line 36: ([SHA1](https://www.apache.org/dist/kudu/%VERSION%/apache-kudu-%VERSION%.tar.gz.sha), > this should be SHA512 now extension is sha512 too instead of sha http://gerrit.cloudera.org:8080/#/c/8800/1/docs/support/scripts/release_index.md.template@36 PS1, Line 36: www same here, link should point to archive.apache.org instead of www.apache.org (KEYS should remain at www though as that can be updated unlike the release files) -- To view, visit http://gerrit.cloudera.org:8080/8800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If55437e0561f0bf8e5e0ec0ae6e5bfca0311fb7d Gerrit-Change-Number: 8800 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Fri, 26 Oct 2018 19:57:18 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > That's one way to address it. I a wonder if there is a way to refactor the This should work without issue in a spark-submit job, I tested that previously. I talked with a colleague on the Spark team, and noted that the proper reference for a standalone spark cluster is simply "spark://:7077" so that should be very easy to implement after all. Let me make those changes, and go back and attempt the same spark-submit test again to make sure this is still working. Could you provide me with the steps you took to submit it, as well as the submit command you used? -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 19:45:39 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > I'm starting to wonder if this is something we should even expose at this p That's one way to address it. I a wonder if there is a way to refactor the code, so that it can be submitted to Spark with spark-submit as well as ran as a java application. What makes me uncomfortable with the current example is that while you can run it locally (w/o a Spark cluster), you cannot run it against a Spark 2.x on Yarn and you cannot run it against Spark 1.x standalone (I tried). So, as a result, you cannot run this example against a Spark cluster deployed with CDH or HDP. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 19:28:06 + Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11801 to look at the new patch set (#2). Change subject: Improve consensus queue overflow logging .. Improve consensus queue overflow logging Suppose tablet server X is a leader of T tablets for which tablet server Y is a follower. The relevant situation is when T is on the order of 100-1000. If Y strains under its consensus load and falls behind processing consensus service requests, UpdateConsensus requests from the leader will get rejected and cause a message to be logged on the leader X for each of the T tablets. The message looks like: W1022 17:20:59.767554 13057 consensus_peers.cc:422] T 9255fdf03ad4451e9fcd62f26741bfe6 P 892cc0d4442c4cdaaf633ed2732f9246 -> Peer dc0af5867d52468f8fd47abf13c08040 (tablet_server_Y.kudu.com:7050): Couldn't send request to peer dc0af5867d52468f8fd47abf13c08040 for tablet 9255fdf03ad4451e9fcd62f26741bfe6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 10.1.1.1:55528 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1 times. Y's consensus service pool also logs the same thing, but it doesn't have the information about the tablet id or peer ids available to it, and it is throttled to occur no more than once per second: W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. This patch attempts to reduce the spam of the first message in the logs by throttling it to occur once every 5 retries. It still is logged for every tablet peer, but those messages are useful if one wants to trace the history of a particular tablet. I also added the throttling messages to Y's output, so it's now W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 5 similar messages] when e.g. 5 other messages have been suppressed. Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc --- M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_peers.h M src/kudu/rpc/service_pool.cc M src/kudu/util/logging.h 4 files changed, 22 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/11801/2 -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc File src/kudu/consensus/consensus_peers.cc: http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc@458 PS1, Line 458: .$2 > How is it going to look like and why is it separated by a dot from the tabl See L446- it's either an empty string or an actual messages depending on the sort of error that happened, so the Substitute string is funny-looking so the actual message is normal-looking in both situations. http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc@461 PS1, Line 461: tablet_id_ > The LOG_WITH_PREFIX_UNLOCKED() produces a string prefix that already contai Done -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 19:15:36 + Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [site] Add http to https redirect
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11162 ) Change subject: [site] Add http to https redirect .. [site] Add http to https redirect In the latest Chrome version all pages without TLS get a "Not Secure" tag in the address bar, so even as there's no secure information (e.g. passwords) transmitted to kudu.apache.org, http should redirect to https. Change-Id: Ic5a060a419466ec4b16840347d387262ca8a4199 Reviewed-on: http://gerrit.cloudera.org:8080/11162 Tested-by: Attila Bukor Reviewed-by: Mike Percy --- M .htaccess 1 file changed, 4 insertions(+), 0 deletions(-) Approvals: Attila Bukor: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11162 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: merged Gerrit-Change-Id: Ic5a060a419466ec4b16840347d387262ca8a4199 Gerrit-Change-Number: 11162 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11772 ) Change subject: Updating web site for Kudu 1.8.0 release .. Updating web site for Kudu 1.8.0 release Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 Reviewed-on: http://gerrit.cloudera.org:8080/11772 Reviewed-by: Grant Henke Tested-by: Grant Henke --- M apidocs M cpp-client-api A docs D docs/administration.html D docs/background_tasks.html D docs/command_line_tools.html D docs/command_line_tools_reference.html D docs/configuration.html D docs/configuration_reference.html D docs/configuration_reference_unsupported.html D docs/contributing.html D docs/developing.html D docs/export_control.html D docs/index.html D docs/installation.html D docs/known_issues.html D docs/kudu-master_configuration_reference.html D docs/kudu-master_configuration_reference_unsupported.html D docs/kudu-tserver_configuration_reference.html D docs/kudu-tserver_configuration_reference_unsupported.html D docs/kudu_impala_integration.html D docs/prior_release_notes.html D docs/quickstart.html D docs/release_notes.html D docs/scaling_guide.html D docs/schema_design.html D docs/security.html D docs/transaction_semantics.html D docs/troubleshooting.html A releases/1.8.0/apidocs/allclasses-frame.html A releases/1.8.0/apidocs/allclasses-noframe.html A releases/1.8.0/apidocs/constant-values.html A releases/1.8.0/apidocs/deprecated-list.html A releases/1.8.0/apidocs/help-doc.html A releases/1.8.0/apidocs/index-all.html A releases/1.8.0/apidocs/index.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.ColumnSchemaBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.CompressionAlgorithm.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.Encoding.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnTypeAttributes.ColumnTypeAttributesBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnTypeAttributes.html A releases/1.8.0/apidocs/org/apache/kudu/Schema.html A releases/1.8.0/apidocs/org/apache/kudu/Type.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-frame.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-summary.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-tree.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-use.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.ColumnSchemaBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.CompressionAlgorithm.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.Encoding.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnTypeAttributes.ColumnTypeAttributesBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnTypeAttributes.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/Schema.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/Type.html A releases/1.8.0/apidocs/org/apache/kudu/client/AbstractKuduScannerBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AlterTableOptions.html A releases/1.8.0/apidocs/org/apache/kudu/client/AlterTableResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduClient.AsyncKuduClientBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduClient.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.AsyncKuduScannerBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.ReadMode.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduSession.html A releases/1.8.0/apidocs/org/apache/kudu/client/ColumnRangePredicate.html A releases/1.8.0/apidocs/org/apache/kudu/client/CreateTableOptions.html A releases/1.8.0/apidocs/org/apache/kudu/client/Delete.html A releases/1.8.0/apidocs/org/apache/kudu/client/DeleteTableResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/ExternalConsistencyMode.html A releases/1.8.0/apidocs/org/apache/kudu/client/HasFailedRpcException.html A releases/1.8.0/apidocs/org/apache/kudu/client/Insert.html A releases/1.8.0/apidocs/org/apache/kudu/client/IsAlterTableDoneResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/IsCreateTableDoneResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduClient.KuduClientBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduClient.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduException.OriginalException.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduException.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduPredicate.ComparisonOp.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduPredicate.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduScanToken.KuduScanTokenBuilder.html A
[kudu-CR](gh-pages) [blog] Add post about 1.8.0 release
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11776 ) Change subject: [blog] Add post about 1.8.0 release .. [blog] Add post about 1.8.0 release Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Reviewed-on: http://gerrit.cloudera.org:8080/11776 Reviewed-by: Grant Henke Tested-by: Grant Henke --- A _posts/2018-10-26-apache-kudu-1-8-0-released.md 1 file changed, 39 insertions(+), 0 deletions(-) Approvals: Grant Henke: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: merged Gerrit-Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Gerrit-Change-Number: 11776 Gerrit-PatchSet: 5 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon
[kudu-CR] Improve consensus queue overflow logging
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc File src/kudu/consensus/consensus_peers.cc: http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc@458 PS1, Line 458: .$2 How is it going to look like and why is it separated by a dot from the tablet id? http://gerrit.cloudera.org:8080/#/c/11801/1/src/kudu/consensus/consensus_peers.cc@461 PS1, Line 461: tablet_id_ The LOG_WITH_PREFIX_UNLOCKED() produces a string prefix that already contains information on tablet identifier. Is it really necessary to duplicate it in the message itself? -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 18:50:34 + Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 1: Yes. KUDU-1707 should help greatly. I'm not working it on it presently but it's on my 3-month list of objectives, let's say. -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 18:49:24 + Gerrit-HasComments: No
[kudu-CR] KUDU-2542: add initial authorization token impl
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/11750 ) Change subject: KUDU-2542: add initial authorization token impl .. Patch Set 3: (1 comment) Just passing through. http://gerrit.cloudera.org:8080/#/c/11750/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11750/3//COMMIT_MSG@19 PS3, Line 19: : The tokens leverage the same token signer as the authentication tokens, : though with the token validity interval configured via the new flag : --authz_token_validity_seconds. I'm curious why you chose to use a new gflag rather than generalizing the existing one. -- To view, visit http://gerrit.cloudera.org:8080/11750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id28747ec38675abdf50dce1e7c176d29213e370f Gerrit-Change-Number: 11750 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 18:36:09 + Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/11801 ) Change subject: Improve consensus queue overflow logging .. Patch Set 1: Code-Review+2 This will become less of an issue when KUDU-1707 is fixed, right? Is that something you're working on? -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Fri, 26 Oct 2018 18:30:53 + Gerrit-HasComments: No
[kudu-CR] KUDU-2542: add initial authorization token impl
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/11750 ) Change subject: KUDU-2542: add initial authorization token impl .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/11750/1/src/kudu/security/token.proto File src/kudu/security/token.proto: http://gerrit.cloudera.org:8080/#/c/11750/1/src/kudu/security/token.proto@49 PS1, Line 49: // Per-column privileges, indexed b > Right, if a user wants to update a single column, they will need update pri FWIW Sentry may eventually get per-column update privileges, it's the most likely next extension for per-column privileges. That's the reason ColumnPrivilegePB isn't scan specific. -- To view, visit http://gerrit.cloudera.org:8080/11750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id28747ec38675abdf50dce1e7c176d29213e370f Gerrit-Change-Number: 11750 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 18:26:41 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster > Could you provide a command-line example with proper SparkMaster URL that p I'm starting to wonder if this is something we should even expose at this point, given the purpose of this example was supposed to be as simple as possible. If we leave this local, there's really no point in exposing it and documenting how to change it. I can work on an example of how to target a remote cluster, but it will require a bit of reworking of the code as we'll need to add some additional values in order to gather the necessary YARN configuration files, etc. which might be more than a simple example like this should need. Thoughts? -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 18:26:12 + Gerrit-HasComments: Yes
[kudu-CR] Improve consensus queue overflow logging
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11801 Change subject: Improve consensus queue overflow logging .. Improve consensus queue overflow logging Suppose tablet server X is a leader of T tablets for which tablet server Y is a follower. The relevant situation is when T is on the order of 100-1000. If Y strains under its consensus load and falls behind processing consensus service requests, UpdateConsensus requests from the leader will get rejected and cause a message to be logged on the leader X for each of the T tablets. The message looks like: W1022 17:20:59.767554 13057 consensus_peers.cc:422] T 9255fdf03ad4451e9fcd62f26741bfe6 P 892cc0d4442c4cdaaf633ed2732f9246 -> Peer dc0af5867d52468f8fd47abf13c08040 (tablet_server_Y.kudu.com:7050): Couldn't send request to peer dc0af5867d52468f8fd47abf13c08040 for tablet 9255fdf03ad4451e9fcd62f26741bfe6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 10.1.1.1:55528 dropped due to backpressure. The service queue is full; it has 50 items.. Retrying in the next heartbeat period. Already tried 1 times. Y's consensus service pool also logs the same thing, but it doesn't have the information about the tablet id or peer ids available to it, and it is throttled to occur no more than once per second: W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. This patch attempts to reduce the spam of the first message in the logs by throttling it to occur once every 5 retries. It still is logged for every tablet peer, but those messages are useful if one wants to trace the history of a particular tablet. I also added the throttling messages to Y's output, so it's now W1022 17:37:33.535168 4330 service_pool.cc:130] UpdateConsensus request on kudu.consensus.ConsensusService from 10.45.26.115:36820 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 5 similar messages] when e.g. 5 other messages have been suppressed. Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc --- M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_peers.h M src/kudu/rpc/service_pool.cc M src/kudu/util/logging.h 4 files changed, 23 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/11801/1 -- To view, visit http://gerrit.cloudera.org:8080/11801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7697c63babefac0f76bcc8c87d70f7e7125e55cc Gerrit-Change-Number: 11801 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] WIP:[sentry] Integrate AuthzProvider into CatalogManager
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/11797 ) Change subject: WIP:[sentry] Integrate AuthzProvider into CatalogManager .. Patch Set 1: (10 comments) http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc File src/kudu/master/catalog_manager.cc: http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@101 PS1, Line 101: #include "kudu/master/default_authz_provider.h" missing authz_provider.h? http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@257 PS1, Line 257: DEFINE_string(trusted_user_acl, "", spitballing: in order to keep as much of the authz behavior in an easily auditable state, it may be good to instead define this flag in authz_provider.cc, then you can add a helper function which checks a username against the ACL without having to do the std::find stuff repeatedly in catalog manager. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@1238 PS1, Line 1238: authz_provider_->Stop(); It should always be initialized, right? http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@1403 PS1, Line 1403: ector Probably won't be an issue, but just to head off any potential issues, consider making this an unordered_set, or instead sorting the vector and using std::binary_search. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@1403 PS1, Line 1403: vecto Since this isn't a runtime flag, consider doing this once and caching it in a static with std::once. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@1813 PS1, Line 1813: RETURN_NOT_OK(FindAndLockTable(req, resp, LockMode::READ, , )); This looks like a potential TOCTOU issue. We're also locking the table on line 1835 in the HMS path, and inside DeleteTable in the non-HMS path. There's a potential that the table could be switched out in-between, in which case the authorization would have applied to a different table. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/catalog_manager.cc@2227 PS1, Line 2227: RETURN_NOT_OK(FindAndLockTable(req, resp, LockMode::READ, , )); Same thing here. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/master.proto File src/kudu/master/master.proto: http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/master.proto@85 PS1, Line 85: NOT_AUTHORIZED = 14; add docs http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/sentry_authz_provider.cc File src/kudu/master/sentry_authz_provider.cc: PS1: These changes look like they should be in the SentryAuthzProvider patch. http://gerrit.cloudera.org:8080/#/c/11797/1/src/kudu/master/sentry_authz_provider.cc@146 PS1, Line 146: equals iequals? -- To view, visit http://gerrit.cloudera.org:8080/11797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab4aa027ae6eb4520db48ce348db552c9feec2a8 Gerrit-Change-Number: 11797 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Fri, 26 Oct 2018 18:19:28 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: SparkMaster Could you provide a command-line example with proper SparkMaster URL that points to a remote Spark Master? -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 18:03:58 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Dan Burkert has posted comments on this change. (
http://gerrit.cloudera.org:8080/6348 )
Change subject: KUDU-1918 Prevent hijacking of scanner IDs
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc@540
PS3, Line 540: json->Set("remote user", scan.remote_user.username());
> woops, this is wrong. You just need to make the corresponding change in sc
Also add an underscore, I think that's the prevailing style for our JSON keys.
--
To view, visit http://gerrit.cloudera.org:8080/6348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4
Gerrit-Change-Number: 6348
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 26 Oct 2018 18:01:57 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Dan Burkert has posted comments on this change. (
http://gerrit.cloudera.org:8080/6348 )
Change subject: KUDU-1918 Prevent hijacking of scanner IDs
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc@540
PS3, Line 540: json->Set("remote user", scan.remote_user.username());
> This is going to break the web UI; there should be a corresponding change i
woops, this is wrong. You just need to make the corresponding change in
scans.mustache. The title of the column is already hardcoded to 'Requestor',
so that won't change.
--
To view, visit http://gerrit.cloudera.org:8080/6348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4
Gerrit-Change-Number: 6348
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 26 Oct 2018 18:01:14 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Dan Burkert has posted comments on this change. (
http://gerrit.cloudera.org:8080/6348 )
Change subject: KUDU-1918 Prevent hijacking of scanner IDs
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_service.cc@1292
PS3, Line 1292: // TODO(awong): consider validating that the scanner was
created by the
I'd suggest going ahead and doing this as a part of this change, unless it's a
large change in its own right.
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tserver_path_handlers.cc@540
PS3, Line 540: json->Set("remote user", scan.remote_user.username());
This is going to break the web UI; there should be a corresponding change in
scans.mustache. I wouldn't change it though, I think 'remote user' is not as
clear as 'requestor' (although I think 'user' might be better than both).
--
To view, visit http://gerrit.cloudera.org:8080/6348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4
Gerrit-Change-Number: 6348
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 26 Oct 2018 18:00:06 +
Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: If running locally (standalone), this must be set to 'local'. > Thanks for the feedback. Do you think this would be better? Yes, I think this is better. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 17:58:05 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: If running locally (standalone), this must be set to 'local'. > This wording creates an impression that this parameter must be set explici Thanks for the feedback. Do you think this would be better? "This defaults to 'local', which is the required value if running the application locally (standalone)" -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 17:53:06 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/6348 ) Change subject: KUDU-1918 Prevent hijacking of scanner IDs .. Patch Set 3: > One of Todd's comments from KUDU-1843 was: > > Caching the original username turns out to be a little tricky, since the > WAL doesn't record the original username, and thus when reconstructing the > request cache during tablet bootstrap we don't have enough information to do > so. I think making the UUIDs unpredictable is probably a better approach. > > That's still an issue, no? My mistake; I had followed KUDU-1918 to KUDU-1843, and didn't realize that the conversation shifted to talking about _writes_ (which are cached in the request cache). Scans aren't cached, so this isn't an issue here. KUDU-1843 remains at large. -- To view, visit http://gerrit.cloudera.org:8080/6348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 Gerrit-Change-Number: 6348 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 17:52:18 + Gerrit-HasComments: No
[kudu-CR] [examples] Add basic Spark example written in Scala
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc File examples/scala/spark-example/README.adoc: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@41 PS4, Line 41: If running locally (standalone), this must be set to 'local'. This wording creates an impression that this parameter must be set explicitly when running locally, but the way the code works (and what the example below shows) is that this parameter defaults to "local" and is not required to be set. -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 17:50:38 + Gerrit-HasComments: Yes
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/11772 ) Change subject: Updating web site for Kudu 1.8.0 release .. Patch Set 6: > I think the issue is small enough we can fix it in the next > release. +1 -- To view, visit http://gerrit.cloudera.org:8080/11772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 Gerrit-Change-Number: 11772 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Anonymous Coward (314) Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 26 Oct 2018 17:30:22 + Gerrit-HasComments: No
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Alexey Serbin has posted comments on this change. (
http://gerrit.cloudera.org:8080/6348 )
Change subject: KUDU-1918 Prevent hijacking of scanner IDs
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_server-test-base.h
File src/kudu/tserver/tablet_server-test-base.h:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_server-test-base.h@125
PS3, Line 125: Status FillNewScanRequest(ReadMode read_mode,
NewScanRequestPB* scan);
Could it be a static method (or at least const)?
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_server-test.cc@3400
PS3, Line 3400: TEST_F(TabletServerTest, TestScannerCheckMatchingUser) {
Is it worth adding a higher-level test to make sure this works as expected? It
seems to be too early at this phase, but after the authz token stuff is there,
it might make sense.
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_server-test.cc@3407
PS3, Line 3407: different user
Does it make sense to add a scenario to verify that without user_credentials
set for the scanner it's not possible to re-use the original scanner?
--
To view, visit http://gerrit.cloudera.org:8080/6348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4
Gerrit-Change-Number: 6348
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 26 Oct 2018 17:28:23 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Adar Dembo has posted comments on this change. (
http://gerrit.cloudera.org:8080/6348 )
Change subject: KUDU-1918 Prevent hijacking of scanner IDs
..
Patch Set 3:
(1 comment)
One of Todd's comments from KUDU-1843 was:
Caching the original username turns out to be a little tricky, since the WAL
doesn't record the original username, and thus when reconstructing the request
cache during tablet bootstrap we don't have enough information to do so. I
think making the UUIDs unpredictable is probably a better approach.
That's still an issue, no?
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/6348/3/src/kudu/tserver/tablet_service.cc@2066
PS3, Line 2066: return Status::NotAuthorized(Substitute("User $0 requested
scanner it doesn't own", requestor));
Why not populate *error_code with something more detailed? Is UNKNOWN_ERROR the
best we can do?
--
To view, visit http://gerrit.cloudera.org:8080/6348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4
Gerrit-Change-Number: 6348
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 26 Oct 2018 17:17:43 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6348 ) Change subject: KUDU-1918 Prevent hijacking of scanner IDs .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/6348/2/src/kudu/tserver/scanners.cc File src/kudu/tserver/scanners.cc: http://gerrit.cloudera.org:8080/#/c/6348/2/src/kudu/tserver/scanners.cc@145 PS2, Line 145:remote_user, > warning: 'remote_user' used after it was moved [bugprone-use-after-move] Done -- To view, visit http://gerrit.cloudera.org:8080/6348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 Gerrit-Change-Number: 6348 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 16:31:45 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Andrew Wong has uploaded a new patch set (#3) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/6348 ) Change subject: KUDU-1918 Prevent hijacking of scanner IDs .. KUDU-1918 Prevent hijacking of scanner IDs This makes the scanner remember its RemoteUser, and ensures that when continuing a scan, the new requestor matches the original requestor. This prevents one user from somehow obtaining a scanner ID from another and then "hijacking" the in-progress scan. Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 --- M src/kudu/tserver/scanners-test.cc M src/kudu/tserver/scanners.cc M src/kudu/tserver/scanners.h M src/kudu/tserver/tablet_server-test-base.cc M src/kudu/tserver/tablet_server-test-base.h M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tserver_path_handlers.cc 8 files changed, 110 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/6348/3 -- To view, visit http://gerrit.cloudera.org:8080/6348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 Gerrit-Change-Number: 6348 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11772 ) Change subject: Updating web site for Kudu 1.8.0 release .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 Gerrit-Change-Number: 11772 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Anonymous Coward (314) Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 26 Oct 2018 16:15:54 + Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] Add post about 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11776 ) Change subject: [blog] Add post about 1.8.0 release .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Gerrit-Change-Number: 11776 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Oct 2018 16:15:48 + Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] Add post about 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11776 ) Change subject: [blog] Add post about 1.8.0 release .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Gerrit-Change-Number: 11776 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Oct 2018 16:15:14 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11772 ) Change subject: Updating web site for Kudu 1.8.0 release .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 Gerrit-Change-Number: 11772 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Anonymous Coward (314) Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 26 Oct 2018 16:15:09 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Hello Mike Percy, Alexey Serbin, Dan Burkert, Jean-Daniel Cryans, Andrew Wong, helifu, Adar Dembo, Anonymous Coward (314), Grant Henke, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11772 to look at the new patch set (#5). Change subject: Updating web site for Kudu 1.8.0 release .. Updating web site for Kudu 1.8.0 release Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 --- M apidocs M cpp-client-api A docs D docs/administration.html D docs/background_tasks.html D docs/command_line_tools.html D docs/command_line_tools_reference.html D docs/configuration.html D docs/configuration_reference.html D docs/configuration_reference_unsupported.html D docs/contributing.html D docs/developing.html D docs/export_control.html D docs/index.html D docs/installation.html D docs/known_issues.html D docs/kudu-master_configuration_reference.html D docs/kudu-master_configuration_reference_unsupported.html D docs/kudu-tserver_configuration_reference.html D docs/kudu-tserver_configuration_reference_unsupported.html D docs/kudu_impala_integration.html D docs/prior_release_notes.html D docs/quickstart.html D docs/release_notes.html D docs/scaling_guide.html D docs/schema_design.html D docs/security.html D docs/transaction_semantics.html D docs/troubleshooting.html A releases/1.8.0/apidocs/allclasses-frame.html A releases/1.8.0/apidocs/allclasses-noframe.html A releases/1.8.0/apidocs/constant-values.html A releases/1.8.0/apidocs/deprecated-list.html A releases/1.8.0/apidocs/help-doc.html A releases/1.8.0/apidocs/index-all.html A releases/1.8.0/apidocs/index.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.ColumnSchemaBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.CompressionAlgorithm.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.Encoding.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnSchema.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnTypeAttributes.ColumnTypeAttributesBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/ColumnTypeAttributes.html A releases/1.8.0/apidocs/org/apache/kudu/Schema.html A releases/1.8.0/apidocs/org/apache/kudu/Type.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-frame.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-summary.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-tree.html A releases/1.8.0/apidocs/org/apache/kudu/backup/package-use.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.ColumnSchemaBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.CompressionAlgorithm.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.Encoding.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnSchema.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnTypeAttributes.ColumnTypeAttributesBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/ColumnTypeAttributes.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/Schema.html A releases/1.8.0/apidocs/org/apache/kudu/class-use/Type.html A releases/1.8.0/apidocs/org/apache/kudu/client/AbstractKuduScannerBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AlterTableOptions.html A releases/1.8.0/apidocs/org/apache/kudu/client/AlterTableResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduClient.AsyncKuduClientBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduClient.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.AsyncKuduScannerBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.ReadMode.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduScanner.html A releases/1.8.0/apidocs/org/apache/kudu/client/AsyncKuduSession.html A releases/1.8.0/apidocs/org/apache/kudu/client/ColumnRangePredicate.html A releases/1.8.0/apidocs/org/apache/kudu/client/CreateTableOptions.html A releases/1.8.0/apidocs/org/apache/kudu/client/Delete.html A releases/1.8.0/apidocs/org/apache/kudu/client/DeleteTableResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/ExternalConsistencyMode.html A releases/1.8.0/apidocs/org/apache/kudu/client/HasFailedRpcException.html A releases/1.8.0/apidocs/org/apache/kudu/client/Insert.html A releases/1.8.0/apidocs/org/apache/kudu/client/IsAlterTableDoneResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/IsCreateTableDoneResponse.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduClient.KuduClientBuilder.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduClient.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduException.OriginalException.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduException.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduPredicate.ComparisonOp.html A releases/1.8.0/apidocs/org/apache/kudu/client/KuduPredicate.html A
[kudu-CR] WIP: Hive Metastore integration docs
Hello Andrew Wong, Hao Hao, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/11798 to review the following change. Change subject: WIP: Hive Metastore integration docs .. WIP: Hive Metastore integration docs Change-Id: I12939c8f2245450ad46898c2050451b090c7ea01 --- A docs/hive_metastore.adoc 1 file changed, 144 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/11798/1 -- To view, visit http://gerrit.cloudera.org:8080/11798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I12939c8f2245450ad46898c2050451b090c7ea01 Gerrit-Change-Number: 11798 Gerrit-PatchSet: 1 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Hao Hao
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6348 ) Change subject: KUDU-1918 Prevent hijacking of scanner IDs .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/6348/1/src/kudu/tserver/scanners.cc File src/kudu/tserver/scanners.cc: http://gerrit.cloudera.org:8080/#/c/6348/1/src/kudu/tserver/scanners.cc@29 PS1, Line 29: #include "kudu/common/scan_spec.h" > warning: #includes are not sorted properly [llvm-include-order] Done http://gerrit.cloudera.org:8080/#/c/6348/1/src/kudu/tserver/tablet_service.cc File src/kudu/tserver/tablet_service.cc: http://gerrit.cloudera.org:8080/#/c/6348/1/src/kudu/tserver/tablet_service.cc@1664 PS1, Line 1664: RETURN_NOT_OK_PREPEND(EncodedKey::DecodeEncodedString(tablet_schema, scanner->arena(), : scan_pb.last_primary_key(), ), : "Failed to decode last primary key"); : // Increment the start key, so we don't return the last row again. : RETURN_NOT_OK_PREPEND(EncodedKey::IncrementEncodedKey(tablet_schema, , scanner->arena()), : "Failed to increment encoded last row key"); : } > should we do this inside of ScannerManager, by making LookupSacnner return Added a TODO for this in KeepAlive, since it's the only other place we'd do this atm -- To view, visit http://gerrit.cloudera.org:8080/6348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 Gerrit-Change-Number: 6348 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Fri, 26 Oct 2018 15:48:36 + Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] Add post about 1.8.0 release
Hello Mike Percy, Grant Henke, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11776 to look at the new patch set (#3). Change subject: [blog] Add post about 1.8.0 release .. [blog] Add post about 1.8.0 release Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 --- A _posts/2018-10-26-apache-kudu-1-8-0-released.md 1 file changed, 39 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/76/11776/3 -- To view, visit http://gerrit.cloudera.org:8080/11776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: newpatchset Gerrit-Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Gerrit-Change-Number: 11776 Gerrit-PatchSet: 3 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon
[kudu-CR] KUDU-1918 Prevent hijacking of scanner IDs
Andrew Wong has uploaded a new patch set (#2) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/6348 ) Change subject: KUDU-1918 Prevent hijacking of scanner IDs .. KUDU-1918 Prevent hijacking of scanner IDs This makes the scanner remember its RemoteUser, and ensures that when continuing a scan, the new requestor matches the original requestor. This prevents one user from somehow obtaining a scanner ID from another and then "hijacking" the in-progress scan. Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 --- M src/kudu/tserver/scanners-test.cc M src/kudu/tserver/scanners.cc M src/kudu/tserver/scanners.h M src/kudu/tserver/tablet_server-test-base.cc M src/kudu/tserver/tablet_server-test-base.h M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tserver_path_handlers.cc 8 files changed, 110 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/6348/2 -- To view, visit http://gerrit.cloudera.org:8080/6348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic91fa0ca471bd674e35aa2f8de3806b88ad4b3b4 Gerrit-Change-Number: 6348 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. ( http://gerrit.cloudera.org:8080/11788 ) Change subject: [examples] Add basic Spark example written in Scala .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala File examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala: http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@34 PS4, Line 34: import spark.implicits._ > It fails to build: This link explains it a bit better: https://stackoverflow.com/questions/39968707/spark-2-0-missing-spark-implicits -- To view, visit http://gerrit.cloudera.org:8080/11788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f Gerrit-Change-Number: 11788 Gerrit-PatchSet: 4 Gerrit-Owner: Mitch Barnett Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mitch Barnett Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 26 Oct 2018 14:52:29 + Gerrit-HasComments: Yes
[kudu-CR] [examples] Add basic Spark example written in Scala
Mitch Barnett has posted comments on this change. (
http://gerrit.cloudera.org:8080/11788 )
Change subject: [examples] Add basic Spark example written in Scala
..
Patch Set 4:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/11788/1/examples/scala/spark-example/README.adoc
File examples/scala/spark-example/README.adoc:
http://gerrit.cloudera.org:8080/#/c/11788/1/examples/scala/spark-example/README.adoc@43
PS1, Line 43: To specify a value at execution time, you'll specify the
parameter name in the '-D' format. For example, to set a different set of
masters for the Kudu cluster from the command line and use a custom table name,
set the property `KuduMasters` to a CSV of the master addresses in the form
`host:port` and add a table name value, as shown:
> I meant the default you are using in your code. For example, by default you
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc
File examples/scala/spark-example/README.adoc:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@43
PS4, Line 43: To specify a value at execution time, you'll specify the
parameter name in the '-D' format. For example, to set a different set of
masters for the Kudu cluster from the command line and use a custom table name,
set the property `KuduMasters` to a CSV of the master addresses in the form
`host:port` and add a table name value, as shown:
> Nit: line breaks.
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala
File
examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@16
PS4, Line 16: val KuduMasters: String =
System.getProperty("KuduMasters","kudu.master1:7051,kudu.master2:7051,kudu.master3:7051")
//kudu master address list
> Can we use localhost:7051 as the default?
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@22
PS4, Line 22: //defining a class that we'll use to insert data into the table
> Nit: periods at the end of comments.
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@27
PS4, Line 27: val logger = LoggerFactory.getLogger(SparkExample.getClass)
> nit: this can be defined at the top of the object.
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@34
PS4, Line 34: import spark.implicits._
> Can you try moving this? It should be fine to be with the other imports.
It fails to build:
[ERROR]
/Users/mbarnett/src/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala:14:
error: not found: object spark
[ERROR] import spark.implicits._
[ERROR]^
[INFO]
[INFO] BUILD FAILURE
because it's using the 'spark' instance we declare above it, which actually
contains the lib.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@39
PS4, Line 39:List(
> nit: This indentation seams level looks like it is using 4 spaces.
Sorry, didn't quite understand this one. Should this be simply indented, or
level?
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@40
PS4, Line 40: StructField(IdCol,IntegerType,false),
> nit: space after commas.
Done
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@48
PS4, Line 48: kc.createTable(TableName, schema, Seq(IdCol), new
CreateTableOptions().setNumReplicas(3).addHashPartitions(List(IdCol).asJava, 3))
> This still has a replication factor of 3.
Missed the change. Completed now.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@71
PS4, Line 71: val upsertUsers = Array(User("newUserA", 1234),
User("userC", ))
> To show the value of an upsert, should we re-use a key that was already ins
Yep, I am repeating a value from above (the id=1234). This will upsert 1234 and
change the name of 'userA' to 'newUserA'.
If you think the change should be more drastic to call more attention to what
the upsert did, let me know. I can have it change to something like
"BrandNewUser" instead.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@84
PS4, Line
[kudu-CR] [examples] Add basic Spark example written in Scala
Grant Henke has posted comments on this change. (
http://gerrit.cloudera.org:8080/11788 )
Change subject: [examples] Add basic Spark example written in Scala
..
Patch Set 4:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/11788/1/examples/scala/spark-example/README.adoc
File examples/scala/spark-example/README.adoc:
http://gerrit.cloudera.org:8080/#/c/11788/1/examples/scala/spark-example/README.adoc@43
PS1, Line 43: To specify a value at execution time, you'll specify the
parameter name in the '-D' format. For example, to set a different set of
masters for the Kudu cluster from the command line and use a custom table name,
set the property `KuduMasters` to a CSV of the master addresses in the form
`host:port` and add a table name value, as shown:
> I didn't see any default listed for this value - it's inclusion is only due
I meant the default you are using in your code. For example, by default you set
SPARK_MASTER to local if it isn't provided.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc
File examples/scala/spark-example/README.adoc:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/README.adoc@43
PS4, Line 43: To specify a value at execution time, you'll specify the
parameter name in the '-D' format. For example, to set a different set of
masters for the Kudu cluster from the command line and use a custom table name,
set the property `KuduMasters` to a CSV of the master addresses in the form
`host:port` and add a table name value, as shown:
Nit: line breaks.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala
File
examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala:
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@16
PS4, Line 16: val KuduMasters: String =
System.getProperty("KuduMasters","kudu.master1:7051,kudu.master2:7051,kudu.master3:7051")
//kudu master address list
Can we use localhost:7051 as the default?
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@22
PS4, Line 22: //defining a class that we'll use to insert data into the table
Nit: periods at the end of comments.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@27
PS4, Line 27: val logger = LoggerFactory.getLogger(SparkExample.getClass)
nit: this can be defined at the top of the object.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@34
PS4, Line 34: import spark.implicits._
Can you try moving this? It should be fine to be with the other imports.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@39
PS4, Line 39:List(
nit: This indentation seams level looks like it is using 4 spaces.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@40
PS4, Line 40: StructField(IdCol,IntegerType,false),
nit: space after commas.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@48
PS4, Line 48: kc.createTable(TableName, schema, Seq(IdCol), new
CreateTableOptions().setNumReplicas(3).addHashPartitions(List(IdCol).asJava, 3))
This still has a replication factor of 3.
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@71
PS4, Line 71: val upsertUsers = Array(User("newUserA", 1234),
User("userC", ))
To show the value of an upsert, should we re-use a key that was already
inserted?
http://gerrit.cloudera.org:8080/#/c/11788/4/examples/scala/spark-example/src/main/scala/org/apache/kudu/examples/SparkExample.scala@84
PS4, Line 84: finally try {
just finally?
--
To view, visit http://gerrit.cloudera.org:8080/11788
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9ba09f0118c054a07b951e241c31d66245c57d3f
Gerrit-Change-Number: 11788
Gerrit-PatchSet: 4
Gerrit-Owner: Mitch Barnett
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Attila Bukor
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mitch Barnett
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Fri, 26 Oct 2018 13:59:50 +
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] Add post about 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11776 ) Change subject: [blog] Add post about 1.8.0 release .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: I99fd23683e7417ca9bc836edf49a7e02e36c6b74 Gerrit-Change-Number: 11776 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 26 Oct 2018 13:45:51 + Gerrit-HasComments: No
[kudu-CR](gh-pages) Updating web site for Kudu 1.8.0 release
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/11772 ) Change subject: Updating web site for Kudu 1.8.0 release .. Patch Set 4: Code-Review+2 I think the issue is small enough we can fix it in the next release. -- To view, visit http://gerrit.cloudera.org:8080/11772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Iafad2ae1cc4ae5700025ea3167d51c6f886a2240 Gerrit-Change-Number: 11772 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Anonymous Coward (314) Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 26 Oct 2018 13:44:38 + Gerrit-HasComments: No
[kudu-CR] WIP:[sentry] Integrate AuthzProvider into CatalogManager
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11797 Change subject: WIP:[sentry] Integrate AuthzProvider into CatalogManager .. WIP:[sentry] Integrate AuthzProvider into CatalogManager Change-Id: Iab4aa027ae6eb4520db48ce348db552c9feec2a8 --- M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master.proto M src/kudu/master/sentry_authz_provider-test.cc M src/kudu/master/sentry_authz_provider.cc M src/kudu/master/sentry_authz_provider.h M src/kudu/sentry/sentry_policy_service.thrift 7 files changed, 98 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/11797/1 -- To view, visit http://gerrit.cloudera.org:8080/11797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iab4aa027ae6eb4520db48ce348db552c9feec2a8 Gerrit-Change-Number: 11797 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao
