I see the hadoop processes are running in the container, and I am using 1.5.0 release.
[root@0a92e78b6832 /]# jps 1748 NameNode 1912 DataNode 2745 Jps 1499 NodeManager 988 WebAppProxyServer 1372 JobHistoryServer 1086 ResourceManager Curl returns following result: http://localhost:57001 curl: (52) Empty reply from server Thanks, Jason From: Evans Ye <evan...@apache.org> Reply-To: "user@bigtop.apache.org" <user@bigtop.apache.org> Date: Friday, February 19, 2021 at 11:37 AM To: "user@bigtop.apache.org" <user@bigtop.apache.org> Subject: Re: docker-provisioner question Yes. the config for docker-compose.yaml is the only thing that needs to be added. Make sure you've the daemons up and running: $ ./docker-hadoop.sh -e 1 bash WARNING: The DOCKER_IMAGE variable is not set. Defaulting to a blank string. WARNING: The MEM_LIMIT variable is not set. Defaulting to a blank string. [root@36eb2f2533c4 /]# jps 2290 Jps 1941 DataNode 1001 WebAppProxyServer 1101 ResourceManager 1389 JobHistoryServer 1518 NodeManager 1775 NameNode If the namenode/resourcemanager are not up. The provisioning might has an issue. Make sure you've switched to branch-1.5 because the master branch is currently in transition upgrading to Hadoop 3 and the provisioner might fail. - Evans Jason Wen <zhenshan....@workday.com<mailto:zhenshan....@workday.com>> 於 2021年2月20日 週六 上午12:18寫道： Hi Evans, Thanks for your reply. I modified docker-compose.yaml as you suggested and run `./gradlew docker-provisioner -Pprefix=1.5.0 -Pnum_instances=3` I can see the port mapping on docker containers: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d97eada3f01d bigtop/puppet:1.5.0-centos-7 "/sbin/init" 9 hours ago Up 9 hours 0.0.0.0:18001->8088/tcp, 0.0.0.0:57001->50070/tcp, 0.0.0.0:57501->50075/tcp 20210218_232241_r25597_bigtop_3 20a7fbadd5f8 bigtop/puppet:1.5.0-centos-7 "/sbin/init" 9 hours ago Up 9 hours 0.0.0.0:18000->8088/tcp, 0.0.0.0:57000->50070/tcp, 0.0.0.0:57500->50075/tcp 20210218_232241_r25597_bigtop_2 0a92e78b6832 bigtop/puppet:1.5.0-centos-7 "/sbin/init" 9 hours ago Up 9 hours 0.0.0.0:18002->8088/tcp, 0.0.0.0:57002->50070/tcp, 0.0.0.0:57502->50075/tcp 20210218_232241_r25597_bigtop_1 However it does not work when I try to access http://localhost:57000<https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A57000&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=cp-l8GE6gVhlReZ92oxBuSBinEwCtGcCCPxO9zZAYgI&s=TQgifr-OVsB3jPD1wjb6lvhvr-a7kbtSz_fIXZapEPw&e=> or http://localhost:57001<https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A57001&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=cp-l8GE6gVhlReZ92oxBuSBinEwCtGcCCPxO9zZAYgI&s=Aog6ht98dRjfWzKdfGEN5p-PRMmUCzh8evfXTJz8ahs&e=> or http://localhost:57002<https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A57002&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=cp-l8GE6gVhlReZ92oxBuSBinEwCtGcCCPxO9zZAYgI&s=mrRp8-npBVxHzqLfhYA9tg8bXOHT8D3Nv32Yw3lChrE&e=> Is there any other step that I missed? Thanks, Jason From: Evans Ye <evan...@apache.org<mailto:evan...@apache.org>> Reply-To: "user@bigtop.apache.org<mailto:user@bigtop.apache.org>" <user@bigtop.apache.org<mailto:user@bigtop.apache.org>> Date: Wednesday, February 17, 2021 at 7:48 PM To: "user@bigtop.apache.org<mailto:user@bigtop.apache.org>" <user@bigtop.apache.org<mailto:user@bigtop.apache.org>> Subject: Re: docker-provisioner question Hi Jason, Out of the box it does not support viewing Web UIs. However, you can easily make that available by adding one line to config port exposing in provisioner's docker-compose.yaml[1]. Try to add this: bigtop: image: ${DOCKER_IMAGE} command: /sbin/init domainname: bigtop.apache.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__bigtop.apache.org&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=-4GOcb1jNZJyz3xuMVFK0x2SMwIuOz7bmR8Trk_rtik&s=vDb_YQxDhJZJ8kj7X1SeVM0QefVztf4Qd6a2HKw7c58&e=> privileged: true mem_limit: ${MEM_LIMIT} volumes: - ../../:/bigtop-home - ./config/hiera.yaml:/etc/puppet/hiera.yaml - ./config/hieradata:/etc/puppet/hieradata - ./config/hosts:/etc/hosts - /sys/fs/cgroup:/sys/fs/cgroup:ro ports: - 57000-57099:50070 - 57500-57599:50075 - 18000-18099:8088 See [2][3] for reference. For example, containers' 50070 are mapped host's 50070~50099. The mapping is automatically done by docker. You can use the -l command to see actual mapping: ./docker-hadoop.sh -l Name Command State Ports -------------------------------------------------------------------------------------------------------------------------------- 20210218_113240_r104_bigtop_1 /sbin/init Up 0.0.0.0:57003->50070/tcp, 0.0.0.0:57503->50075/tcp, 0.0.0.0:18003->8088/tcp 20210218_113240_r104_bigtop_2 /sbin/init Up 0.0.0.0:57002->50070/tcp, 0.0.0.0:57502->50075/tcp, 0.0.0.0:18002->8088/tcp I think it would be great to open up a PR to add the feature after you have it tested on your end. What do you think? Best, Evans Ye [1] https://github.com/apache/bigtop/blob/master/provisioner/docker/docker-compose.yml<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_bigtop_blob_master_provisioner_docker_docker-2Dcompose.yml&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=-4GOcb1jNZJyz3xuMVFK0x2SMwIuOz7bmR8Trk_rtik&s=Bn4eJbjimODq_lblNY-w0G18jsUFlV5KhyYwRJ-dCpA&e=> [2] https://docs.docker.com/engine/reference/run/#expose-incoming-ports<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.docker.com_engine_reference_run_-23expose-2Dincoming-2Dports&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=-4GOcb1jNZJyz3xuMVFK0x2SMwIuOz7bmR8Trk_rtik&s=b-xqfN5yad-JycWdF2rfroo18KjeE24vBaqWITE4wD4&e=> [3] https://github.com/compose-spec/compose-spec/blob/master/spec.md#ports<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_compose-2Dspec_compose-2Dspec_blob_master_spec.md-23ports&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=-4GOcb1jNZJyz3xuMVFK0x2SMwIuOz7bmR8Trk_rtik&s=hg4Lg2RWu4vLvigN1O1Fq0IEoGzGSa_4zUXg2E8aJKQ&e=> Jason Wen <zhenshan....@workday.com<mailto:zhenshan....@workday.com>> 於 2021年2月18日 週四 上午12:44寫道： Hi, I am following https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_BIGTOP_Quickstart-2BGuide-253A-2BBigtop-2BIntegration-2BTest-2BFramework-2B2.0&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=-4GOcb1jNZJyz3xuMVFK0x2SMwIuOz7bmR8Trk_rtik&s=UgN_KP5zcmTmhk70LyUTB1xgvE_4NHcuz5QL9QRVzlg&e=> to use bigtop. I have deployed a Hadoop cluster successfully using `./gradlew docker-provisioner`. How can I access the Namenode/ResourceManager web UI from my local machine? Thanks, Jason
