uranusjr commented on a change in pull request #16954:
URL: https://github.com/apache/airflow/pull/16954#discussion_r670217140
##########
File path: airflow/utils/python_virtualenv.py
##########
@@ -35,12 +38,44 @@ def _generate_virtualenv_cmd(tmp_dir: str, python_bin: str,
system_site_packages
return cmd
-def _generate_pip_install_cmd(tmp_dir: str, requirements: List[str]) ->
Optional[List[str]]:
+def _generate_pip_install_cmd(tmp_dir: str,
+ requirements: List[str],
+ connection_id: Optional[str] = None
+ ) -> Optional[List[str]]:
if not requirements:
return None
- # direct path alleviates need to activate
- cmd = [f'{tmp_dir}/bin/pip', 'install']
- return cmd + requirements
+
+ if connection_id:
+ con: Connection = BaseHook.get_connection(connection_id)
+ user = con.login
+ schema = con.schema or 'https'
+ password = con.get_password()
+ port = con.port or ''
+ host = con.host
+ path = con.extra_dejson.get('path', '')
+ pypi_as_fallback = con.extra_dejson.get('pypi_as_fallback', False)
+
+ if user:
+ netloc = f'{user}:{password}@{host}'
+ else:
+ netloc = host
+
+ netloc = netloc + f':{port}' if port else netloc
+
+ index_url = SplitResult(scheme=schema,
+ netloc=netloc, path=path, query='',
fragment='').geturl()
+
+ private_cmd = [f'{tmp_dir}/bin/pip',
+ 'install',
+ f'--index-url', index_url]
+
+ if pypi_as_fallback:
+ private_cmd.extend([f'--extra-index-url',
'https://pypi.org/simple'])
Review comment:
In general, when your organisaion needs `--extra-index-url`, you don’t
really need it and are identifying the issue wrong 🙂 There are exceptions, e.g.
you are deploying to exotic hardware that PyPI doesn’t have wheels for, and you
want to supply wheels for those PyPI project. For those exceptional use cases,
setting `--extra-index-url` (while keeping the main index URL pointing to PyPI)
is the right thing to do. But if what you need is to install a package not on
PyPI, `--extra-index-url` is the wrong solution.
The suggestion is, instead of having multiple indexes serving different
packages, making all requests go through the private index server you control.
The index is configured in a way that when your private package is requeted,
serve your own wheels, and when a PyPI package is requested, forward the
request to the actual PyPI. This mechanism is implemented by Python index tools
such as devpi (called index inheritance). I heard Artifactory can do that as
well, although I’ve never personally understood how Artifactory works and can’t
confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
