[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha Reviewed-on: http://gerrit.cloudera.org:8080/17125 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 137 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 14 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 13 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 22:46:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6938/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 13 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 17:07:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 12 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 17:07:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 13 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 17:07:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6933/ -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 03 Mar 2021 02:21:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8293/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 12 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:32:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Kurt Deschler has uploaded a new patch set (#12) to the change originally created by John Sherman. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 137 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/12 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 12 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8290/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 11 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 21:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Kurt Deschler has uploaded a new patch set (#11) to the change originally created by John Sherman. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 136 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/11 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 11 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:38:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6933/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 10 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:38:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 20:37:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8286/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 17:17:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, Thomas Tauber-Marshall, Kurt Deschler, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#9). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 137 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/9 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 9 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 8: I'll investigate the failure(s). Initial look was it is unit test - which I should have ran locally in the first place (but evidently did not). -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 8 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 15:20:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6927/ -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 7 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 04:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8276/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 8 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 02 Mar 2021 01:53:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Kurt Deschler has uploaded a new patch set (#8) to the change originally created by John Sherman. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 136 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/8 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 8 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8272/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 6 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 22:47:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 7 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 22:40:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6927/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 7 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 22:40:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 6: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h File be/src/rpc/authentication.h: http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h@67 PS5, Line 67: : AuthProvider* GetEx > I totally did - usually I self review a little better. Thanks for catching. No worries -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 6 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 22:37:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h File be/src/rpc/authentication.h: http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h@67 PS5, Line 67: Currently this is either null if external_fe_port <= 0 or : /// NoAuthProvider. > I think you put this above the wrong function? I totally did - usually I self review a little better. Thanks for catching. http://gerrit.cloudera.org:8080/#/c/17125/5/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/17125/5/bin/start-impala-cluster.py@235 PS5, Line 235: 'external_fe_port': DEFAULT_EXTERNAL_FE_PORT + instance_num, > What would you think about putting this behind a flag for now, eg. "start-i Done I went with the approach of excluding it in build_impalad_port_args since that seemed like a nice clean way of excluding it even if the map contains the port. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 22:26:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, Thomas Tauber-Marshall, Kurt Deschler, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#6). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - Added enable_external_fe_support option to start-impala-cluster.py - which when detected will start impala clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 136 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/6 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 6 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: (2 comments) > > (2 comments) > > > > Are there any existing hs2 methods that it might make sense to > > block for the "external frontend" server? eg. we might want to > > return an error for ExecuteStatement() from it if the external > > frontend will never need to call it. > > > > That would give us a little more safety in case people > accidentally > > expose this port to the outside world (of course, it would still > be > > possible for bad actors to use the ExecutePlannedStatement > > interface, but its probably a lot harder to put together a valid > > TExecRequest to use it than it is to put together a SQL string > like > > ExecuteStatement takes) > > > > Also out of curiosity - what's the long run testing plan here? > Are > > we going to have an actual external FE running in the minicluster > > that can exercise this stuff? > > So the current implementation of external frontend does utilize the > ExecuteStatement functionality (for things like COMPUTE STATS). I > do agree with your assessment that it would be nice to reduce the > surface area in the future. The long term plan would also likely > include enabling similar protections that intra-impalad > communication use between nodes (that prevent people connecting > easily to the backend port and pretending to be a coordinator). > > One option I considered based on your comment was to add a 2nd flag > that would be named something like: external_fe_allow_unsafe which > defaulted to false and disallowed ExecuteStatement via the > external_fe_port. So a user would have to enable external_fe_port > AND set external_fe_allow_unsafe to true to be able to call > ExecuteStatement. But if someone is enabling the external_fe_port - > it is somewhat assumed they know what they are doing so I'm not > 100% convinced this approach is worth it. I am open to suggestions > (or if you like the idea of the 2nd flag). Agreed that sounds unnecessarily complicated. I think its fine as-is for now, just something to keep in mind. > > As for testing - I do believe once the various external FE commits > land we should focus on: > 1) auditing and shoring up what we can build unit tests around > 2) And, yes, it is my understanding that we will eventually be > including an external frontend in the minicluster for more > end-to-end testing. Otherwise, we will need to mock up some sort of > "send pre-made exec request" and "check response" test framework > but I suspect that might be not fun to implement cleanly. Sounds good http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h File be/src/rpc/authentication.h: http://gerrit.cloudera.org:8080/#/c/17125/5/be/src/rpc/authentication.h@67 PS5, Line 67: Currently this is either null if external_fe_port <= 0 or : /// NoAuthProvider. I think you put this above the wrong function? http://gerrit.cloudera.org:8080/#/c/17125/5/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/17125/5/bin/start-impala-cluster.py@235 PS5, Line 235: 'external_fe_port': DEFAULT_EXTERNAL_FE_PORT + instance_num, What would you think about putting this behind a flag for now, eg. "start-impala-cluster.py --enable_external_fe_support" or similar? -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 01 Mar 2021 20:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
John Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: For clarity - for now it'll be the responsibility of external frontends to ensure they work with impala and not vice-versa until we get more stable. We should take due care not to regress any of the current impala functionality and near term unit test what we can - such as not exposing ExecutePlannedStatement via hs2_port and so forth. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: John Sherman Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 01:29:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
j...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: > (2 comments) > > Are there any existing hs2 methods that it might make sense to > block for the "external frontend" server? eg. we might want to > return an error for ExecuteStatement() from it if the external > frontend will never need to call it. > > That would give us a little more safety in case people accidentally > expose this port to the outside world (of course, it would still be > possible for bad actors to use the ExecutePlannedStatement > interface, but its probably a lot harder to put together a valid > TExecRequest to use it than it is to put together a SQL string like > ExecuteStatement takes) > > Also out of curiosity - what's the long run testing plan here? Are > we going to have an actual external FE running in the minicluster > that can exercise this stuff? So the current implementation of external frontend does utilize the ExecuteStatement functionality (for things like COMPUTE STATS). I do agree with your assessment that it would be nice to reduce the surface area in the future. The long term plan would also likely include enabling similar protections that intra-impalad communication use between nodes (that prevent people connecting easily to the backend port and pretending to be a coordinator). One option I considered based on your comment was to add a 2nd flag that would be named something like: external_fe_allow_unsafe which defaulted to false and disallowed ExecuteStatement via the external_fe_port. So a user would have to enable external_fe_port AND set external_fe_allow_unsafe to true to be able to call ExecuteStatement. But if someone is enabling the external_fe_port - it is somewhat assumed they know what they are doing so I'm not 100% convinced this approach is worth it. I am open to suggestions (or if you like the idea of the 2nd flag). As for testing - I do believe once the various external FE commits land we should focus on: 1) auditing and shoring up what we can build unit tests around 2) And, yes, it is my understanding that we will eventually be including an external frontend in the minicluster for more end-to-end testing. Otherwise, we will need to mock up some sort of "send pre-made exec request" and "check response" test framework but I suspect that might be not fun to implement cleanly. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 01:00:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8257/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 27 Feb 2021 00:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#5). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - start-impala-cluster.py starts default clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 134 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/5 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 4: (2 comments) Are there any existing hs2 methods that it might make sense to block for the "external frontend" server? eg. we might want to return an error for ExecuteStatement() from it if the external frontend will never need to call it. That would give us a little more safety in case people accidentally expose this port to the outside world (of course, it would still be possible for bad actors to use the ExecutePlannedStatement interface, but its probably a lot harder to put together a valid TExecRequest to use it than it is to put together a SQL string like ExecuteStatement takes) Also out of curiosity - what's the long run testing plan here? Are we going to have an actual external FE running in the minicluster that can exercise this stuff? http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/rpc/authentication.h File be/src/rpc/authentication.h: http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/rpc/authentication.h@73 PS4, Line 73: AuthProvider* GetExternalFrontendAuthProvider(); Might mention that this is always a NoAuthProvider, or null if external_fe_port is 0 http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/4/be/src/service/impala-server.cc@151 PS4, Line 151: "If 0 or less, the External Frontend server is not started."); You should mention that this server is never protected with authentication so users should be careful to not expose this port if they enable it. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 21:19:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
j...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc@2877 PS3, Line 2877: << external_fe_server_->port(); > tab used for whitespace I'll fix with the review comments. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 26 Feb 2021 18:08:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8245/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 25 Feb 2021 22:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/3/be/src/service/impala-server.cc@2877 PS3, Line 2877: << external_fe_server_->port(); tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 25 Feb 2021 22:30:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17125 to look at the new patch set (#3). Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - start-impala-cluster.py starts default clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 130 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/3 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8244/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 25 Feb 2021 22:27:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17125 ) Change subject: IMPALA-10550: Add External Frontend service port .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/17125/2/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17125/2/be/src/service/impala-server.cc@2876 PS2, Line 2876: LOG(INFO) << "Impala External Frontend Service listening on " << external_fe_server_->port(); line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/17125/2/be/src/service/impalad-main.cc File be/src/service/impalad-main.cc: http://gerrit.cloudera.org:8080/#/c/17125/2/be/src/service/impalad-main.cc@87 PS2, Line 87: Status status = impala_server->Start(FLAGS_beeswax_port, FLAGS_hs2_port, FLAGS_hs2_http_port, line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 25 Feb 2021 22:10:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10550: Add External Frontend service port
Hello Aman Sinha, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/17125 to review the following change. Change subject: IMPALA-10550: Add External Frontend service port .. IMPALA-10550: Add External Frontend service port - If external_fe_port flag is >0, spins up a new HS2 compatible service port - start-impala-cluster.py starts default clusters with external_fe_port on 21150-21152 - Modify impalad_coordinator Dockerfile to expose external frontend port at 21150 - The intent of this commit is to separate external frontend connections from normal hs2 connections - This allows different security policy to be applied to each type of connection. The external_fe_port should be considered a privileged service and should only be exposed to an external frontend that does user authentication and does authorization checks on generated plans Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Reviewed-by: Aman Sinha --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M bin/start-impala-cluster.py M common/thrift/metrics.json M docker/impalad_coordinator/Dockerfile M tests/common/impala_cluster.py 10 files changed, 129 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/17125/2 -- To view, visit http://gerrit.cloudera.org:8080/17125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I991b5b05e12e37d8739e18ed1086bbb0228acc40 Gerrit-Change-Number: 17125 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward
