Re: Private: Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
Re-adding the list. On Wed, 2022-02-02 at 11:51 -0800, Sourabh Hegde wrote: > Hi @Nicolas, Hi Sourabh > Thanks for the detailed answer. > > I followed your steps to fix ssh "config" file and now it's working > fine. Glad to hear it works now. > Regarding my build environment: I am using Docker Desktop for Windows > so i am starting container from Docker Desktop GUI, not from Windows > powershell. So I was confused how to pass "-v > $SSH_AUTH_SOCK:/ssh.socket \". I hope it's making sense here. Please > do let me know if I am doing anything wrong here so that I don't face > similar issues in future. I see, I never used Docker Desktop for Windows, so I can't give you advice about that specifically. If the ssh connection works correctly with the configuration you created, you shouldn't need to forward the ssh agent. > Thanks in advance. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56054): https://lists.yoctoproject.org/g/yocto/message/56054 Mute This Topic: https://lists.yoctoproject.org/mt/88879520/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
On Mon, 2022-01-31 at 02:54 -0800, Sourabh Hegde wrote: > Hello @Nicolas @Erik @Khem, Hi! > Update from my side: > > After following some discussion from other posts, I added "config" > file. > I think you're starting to mix various things together, you should maybe try to not do everything at the same time. I added comments about what is wrong with your config, but depending on your build environment, the ssh config is maybe not the best choice. > ~/.ssh/config: > > Host git.example.com > HostName git.example.com > User git > PreferredAuthentications publickey > IdentityFile ~/.ssh/id_ed25519.pub > # LogLevel DEBUG3 > You need to specify the private key with IdentityFile, not the public key. > Then I did "eval `ssh-agent -s`" > > Then doing "ssh-add ~/.ssh/id_ed25519.pub" results in: Same here, you should be doing "ssh-add ~/.ssh/id_ed25519" (without the .pub). > @@@ > @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ > @@@ > Permissions 0644 for '/root/.ssh/id_ed25519.pub' are too open. > It is required that your private key files are NOT accessible by > others. > This private key will be ignored. > > Whereas the permissions are set as: > > ls -l -a ~/.ssh > > -rw-r--r-- 1 root root 157 Jan 31 10:48 config > -rw--- 1 root root 464 Jan 20 15:26 id_ed25519 > -rw-r--r-- 1 root root 109 Jan 20 15:26 id_ed25519.pub > -rw-r--r-- 1 root root 888 Jan 26 08:43 known_hosts > Well, the permissions on id_ed25519 are correct, but you added the public key as private key in your config / in your ssh-add command, which doesn't have the required permissions for private keys (because it's not). > "ssh-agent" is running > > ssh-agent > SSH_AUTH_SOCK=/tmp/ssh-lcft54A4nriC/agent.2833; export SSH_AUTH_SOCK; > SSH_AGENT_PID=2834; export SSH_AGENT_PID; > echo Agent pid 2834; > > After doing these changes, when I try to "ssh -v git.example.com" to > test the connection before running bitbake, I get > > OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020 > debug1: Reading configuration data /root/.ssh/config > debug1: /root/.ssh/config line 1: Applying options for > git.example.com > debug1: Reading configuration data /etc/ssh/ssh_config > debug1: /etc/ssh/ssh_config line 19: include > /etc/ssh/ssh_config.d/*.conf matched no files > debug1: /etc/ssh/ssh_config line 21: Applying options for * > debug1: Connecting to git.example.com [116.203.241.xxx] port 22. > debug1: connect to address 116.203.241.xxx port 22: Connection > refused > ssh: connect to host git.example.com port 22: Connection refused > > I don't understand what is the issue here. > > @Nicolas Can you please let me know where and how to run below > commands? Do I need to run them every time before fetching from > gitlab? > -v $SSH_AUTH_SOCK:/ssh.socket \ > -e SSH_AUTH_SOCK=/ssh.socket \ > I think you should explain your build environment a bit better, as I can just guess what you're doing. You should add these parameters when starting your docker container. For example I use something along these lines: docker run -ti --rm -v ~/development/oe-build:/workdir -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK -e SSH_AUTH_SOCK="$SSH_AUTH_SOCK" crops/poky --workdir=/workdir If you're forwarding the ssh agent like this, you don't need a key or config file at all, only known_hosts. On the other hand, if you're using e.g. GitLab pipelines with docker, you should not do it like mentioned above, but follow their guide [1]. [1]: https://docs.gitlab.com/ee/ci/ssh_keys/index.html#ssh-keys-when-using-the-docker-executor > And also I already have "known_hosts" file with matching entries for > key/agent pair. > > Can you please let me know how to make this working? > > Your help will be much appreciated. > > Thanks in advance. > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56043): https://lists.yoctoproject.org/g/yocto/message/56043 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
Hello @Nicolas @Erik @Khem, Update from my side: After following some discussion from other posts, I added "config" file. ~/.ssh/config: Host git.example.com HostName git.example.com User git PreferredAuthentications publickey IdentityFile ~/.ssh/id_ed25519.pub # LogLevel DEBUG3 Then I did "eval `ssh-agent -s`" Then doing "ssh-add ~/.ssh/id_ed25519.pub" results in: @@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@ Permissions 0644 for '/root/.ssh/id_ed25519.pub' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. Whereas the permissions are set as: ls -l -a ~/.ssh -rw-r--r-- 1 root root 157 Jan 31 10:48 config -rw--- 1 root root 464 Jan 20 15:26 id_ed25519 -rw-r--r-- 1 root root 109 Jan 20 15:26 id_ed25519.pub -rw-r--r-- 1 root root 888 Jan 26 08:43 known_hosts "ssh-agent" is running ssh-agent SSH_AUTH_SOCK=/tmp/ssh-lcft54A4nriC/agent.2833; export SSH_AUTH_SOCK; SSH_AGENT_PID=2834; export SSH_AGENT_PID; echo Agent pid 2834; After doing these changes, when I try to "ssh -v git.example.com" to test the connection before running bitbake, I get OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /root/.ssh/config debug1: /root/.ssh/config line 1: Applying options for git.example.com debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to git.example.com [116.203.241.xxx] port 22. debug1: connect to address 116.203.241.xxx port 22: Connection refused ssh: connect to host git.example.com port 22: Connection refused I don't understand what is the issue here. @Nicolas Can you please let me know where and how to run below commands? Do I need to run them every time before fetching from gitlab? -v $SSH_AUTH_SOCK:/ssh.socket \ -e SSH_AUTH_SOCK=/ssh.socket \ And also I already have "known_hosts" file with matching entries for key/agent pair. Can you please let me know how to make this working? Your help will be much appreciated. Thanks in advance. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56028): https://lists.yoctoproject.org/g/yocto/message/56028 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
On Fri, Jan 28, 2022 at 2:27 AM VIVAVIS AG wrote: > > Hi, > > > Von: yocto@lists.yoctoproject.org Im Auftrag > > von Sourabh Hegde > > Gesendet: Freitag, 28. Januar 2022 10:47 > > > > Can you please let me know how to "forward SSH_AGENT into it to be able > > to fetch from internal projects without the need to mount the key into the > > container."? I never did that before. > > I use the following options within the Docker run command: > -v $SSH_AUTH_SOCK:/ssh.socket \ > -e SSH_AUTH_SOCK=/ssh.socket \ > > Furthermore, I had to mount the .ssh folder into the container to make it > working (be aware of security risk). > Additionally, you should check that uid, gid of the user in the container is > the same on the host. > yeah something like that works, we use it for yoe which always uses container to build see https://github.com/YoeDistro/yoe-distro/blob/master/envsetup.sh#L528-L541 > Regards, > > Carsten > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56013): https://lists.yoctoproject.org/g/yocto/message/56013 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
On Fri, Jan 28, 2022 at 11:50 AM Nicolas Jeker wrote: > > On Fri, 2022-01-28 at 10:27 +, VIVAVIS AG wrote: > > Hi, > > > > > Von: yocto@lists.yoctoproject.org Im > > > Auftrag von Sourabh Hegde > > > Gesendet: Freitag, 28. Januar 2022 10:47 > > > > > > Can you please let me know how to "forward SSH_AGENT into it to be > > > able > > > to fetch from internal projects without the need to mount the key > > > into the container."? I never did that before. > > > > I use the following options within the Docker run command: > > -v $SSH_AUTH_SOCK:/ssh.socket \ > > -e SSH_AUTH_SOCK=/ssh.socket \ > > > > That's pretty much what I use. > > > Furthermore, I had to mount the .ssh folder into the container to > > make it working (be aware of security risk). > > Additionally, you should check that uid, gid of the user in the > > container is the same on the host. > > I do something similar, my "problem" was that ssh needs the > .ssh/known_hosts file with a matching entry in addition to your > key/agent, but mounting the .ssh folder was not possible for me because > of permissions. Currently, I just created a little script that wraps > "oe-init-build-env" and populates the known_hosts file accordingly. > > mkdir -p ~/.ssh > > cat <> ~/.ssh/known_hosts > git.example.com ssh-ed25519 > EOF > I use my own Dockerfile based on crops/poky where I do the following, which might be helpful if you also use this. It sets up the config changes in /etc/skel/ since it creates users "on the fly" with matching uid. # Remove strict host key checking for ssh # This is needed since the build will pull source over git-ssh RUN mkdir -p /etc/skel/.ssh/ COPY ci-scripts/docker-stuff/config /etc/skel/.ssh/ RUN echo 'export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"' >> /etc/skel/.bashrc The ci-scripts/docker-stuff/config file contains: Host * StrictHostKeyChecking no UserKnownHostsFile=/dev/null Now it was ages ago I set this up, and right now I can't really understand why I basically do the same thing twice. So you'd have to check which of the two things that actually solves the issue :-) Cheers, Erik > > Regards, > > > > Carsten > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56010): https://lists.yoctoproject.org/g/yocto/message/56010 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
On Fri, 2022-01-28 at 10:27 +, VIVAVIS AG wrote: > Hi, > > > Von: yocto@lists.yoctoproject.org Im > > Auftrag von Sourabh Hegde > > Gesendet: Freitag, 28. Januar 2022 10:47 > > > > Can you please let me know how to "forward SSH_AGENT into it to be > > able > > to fetch from internal projects without the need to mount the key > > into the container."? I never did that before. > > I use the following options within the Docker run command: > -v $SSH_AUTH_SOCK:/ssh.socket \ > -e SSH_AUTH_SOCK=/ssh.socket \ > That's pretty much what I use. > Furthermore, I had to mount the .ssh folder into the container to > make it working (be aware of security risk). > Additionally, you should check that uid, gid of the user in the > container is the same on the host. I do something similar, my "problem" was that ssh needs the .ssh/known_hosts file with a matching entry in addition to your key/agent, but mounting the .ssh folder was not possible for me because of permissions. Currently, I just created a little script that wraps "oe-init-build-env" and populates the known_hosts file accordingly. mkdir -p ~/.ssh cat <> ~/.ssh/known_hosts git.example.com ssh-ed25519 EOF > Regards, > > Carsten > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56009): https://lists.yoctoproject.org/g/yocto/message/56009 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
Hi, > Von: yocto@lists.yoctoproject.org Im Auftrag > von Sourabh Hegde > Gesendet: Freitag, 28. Januar 2022 10:47 > > Can you please let me know how to "forward SSH_AGENT into it to be able > to fetch from internal projects without the need to mount the key into the > container."? I never did that before. I use the following options within the Docker run command: -v $SSH_AUTH_SOCK:/ssh.socket \ -e SSH_AUTH_SOCK=/ssh.socket \ Furthermore, I had to mount the .ssh folder into the container to make it working (be aware of security risk). Additionally, you should check that uid, gid of the user in the container is the same on the host. Regards, Carsten -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56008): https://lists.yoctoproject.org/g/yocto/message/56008 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
Hi Nicolas, Thanks for your answer. That's great. Even I am building inside a docker container. I tried with creating a "config" file in .ssh directory. But I still have same issue. Can you please let me know how to "forward SSH_AGENT into it to be able to fetch from internal projects without the need to mount the key into the container."? I never did that before. Thanks in advance. On Fri, Jan 28, 2022, 10:42 Nicolas Jeker wrote: > On Tue, 2022-01-25 at 23:16 -0800, hrsourabh...@gmail.com wrote: > > I am trying to fetch a private gitlab repo within Yocto image recipe > > using SSH protocol. In my image recipe I have passed SRC_URI as: > > > > SRC_URI = " \ > > gitsm://g...@git.example.com:2224/blah/blah/blah/blah;protocol > > =ssh;branch=master \ > > " > > I use almost the same, just without submodules. > > SRC_URI = > "git://g...@git.example.com:1234/group/project.git;protocol=ssh" > > It should "just work" if ssh is able to find your key. I often build in > a docker container, so I have to forward SSH_AGENT into it to be able > to fetch from internal projects without the need to mount the key into > the container. I don't really have any insight for builds outside > docker, if git clone works, the bitbake fetcher should too. > > > But this results in the error: > > > > > > > But I am able to clone the repo using git clone. > > SSH key is already added to the Gitlab. There is no config file in my > > ~/.ssh. Do I need to create a config file? What should be the content > > of the config file? > > You should not need a ssh config file. > > > Can anyone please let me know how to resolve this? > > Thanks in advance. > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56007): https://lists.yoctoproject.org/g/yocto/message/56007 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake
On Tue, 2022-01-25 at 23:16 -0800, hrsourabh...@gmail.com wrote: > I am trying to fetch a private gitlab repo within Yocto image recipe > using SSH protocol. In my image recipe I have passed SRC_URI as: > > SRC_URI = " \ > gitsm://g...@git.example.com:2224/blah/blah/blah/blah;protocol > =ssh;branch=master \ > " I use almost the same, just without submodules. SRC_URI = "git://g...@git.example.com:1234/group/project.git;protocol=ssh" It should "just work" if ssh is able to find your key. I often build in a docker container, so I have to forward SSH_AGENT into it to be able to fetch from internal projects without the need to mount the key into the container. I don't really have any insight for builds outside docker, if git clone works, the bitbake fetcher should too. > But this results in the error: > > > But I am able to clone the repo using git clone. > SSH key is already added to the Gitlab. There is no config file in my > ~/.ssh. Do I need to create a config file? What should be the content > of the config file? You should not need a ssh config file. > Can anyone please let me know how to resolve this? > Thanks in advance. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#56006): https://lists.yoctoproject.org/g/yocto/message/56006 Mute This Topic: https://lists.yoctoproject.org/mt/88691891/21656 Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
