[jira] [Commented] (AIRFLOW-6242) Introduce run_type in DAG Run table
[ https://issues.apache.org/jira/browse/AIRFLOW-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092962#comment-17092962 ] Will Hudgins commented on AIRFLOW-6242: --- [~kamil.bregula] Would I be correct in assuming this might require an AIP? > Introduce run_type in DAG Run table > --- > > Key: AIRFLOW-6242 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6242 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Hello, > Before you start reading this ticket, I want to tell you one thing - you are > wonderful. > We use queries that use like expressions to filter two types of DAG Run. > * Backfill: > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/jobs/backfill_job.py#L52] > * Scheduled > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/models/dagrun.py#L46] > This type of expression can affect performance and it seems a better idea to > store this information in a separate column. This may allow a query planner > to create a better query plan in the database engine. > Thanks for reading this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6458) Deprecation warning about dropping support for Python 2
[
https://issues.apache.org/jira/browse/AIRFLOW-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092961#comment-17092961
]
Will Hudgins commented on AIRFLOW-6458:
---
[~kamil.bregula] If you don't mind, I can work on this one.
> Deprecation warning about dropping support for Python 2
> ---
>
> Key: AIRFLOW-6458
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6458
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>
> We should add a deprecation warning in Airflow 1.10.x to inform users that
> Python 3 is required in the Airflow 2.0.
> We should add something similar to {{airflow/__init__.py}} file
> {code:python}
> if sys.version_info[0] < 3:
> message = (
> "A future version of this application will drop support for Python
> 2.7."
> "More details about Python 2 support for Airflow can be found at "
> "https://cloud.google.com/python/docs/python2-sunset/;
> )
> warnings.warn(message, DeprecationWarning)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092279#comment-17092279 ] Will Hudgins commented on AIRFLOW-6281: --- [~kamil.bregula] PR [https://github.com/apache/airflow/pull/8442] is raised and ready for review :) > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081994#comment-17081994 ] Will Hudgins edited comment on AIRFLOW-6281 at 4/13/20, 2:44 AM: - Read all the supporting material and guidelines, started on writing this one. was (Author: wkhudgins): Started on writing this one. > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081994#comment-17081994 ] Will Hudgins commented on AIRFLOW-6281: --- Started on writing this one. > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076840#comment-17076840
]
Will Hudgins commented on AIRFLOW-6515:
---
[~toopt4] I raised a PR for this: [https://github.com/apache/airflow/pull/8170]
Some of the matches I believed were false positives. Sometimes it was hard to
determine what should be WARN vs ERROR. If you have a chance to review and
think I made some wrong changes or should reexamine the remaining matches,
please let me know.
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: Will Hudgins
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
>
[jira] [Assigned] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Will Hudgins reassigned AIRFLOW-6515:
-
Assignee: Will Hudgins
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: Will Hudgins
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("""Missing
> attribute "%s" when looked-up in Ldap database.
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Parsing error
> when retrieving the user's group(s)."
> airflow/contrib/utils/sendgrid.py:log.warning('Failed to send out
> email with subject %s, status
[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072796#comment-17072796
]
Will Hudgins commented on AIRFLOW-6515:
---
Great, I'll work on this then unless there's any objection [~toopt4]
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("""Missing
> attribute "%s" when looked-up in Ldap database.
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Parsing error
> when retrieving the user's group(s)."
> airflow/contrib/utils/sendgrid.py:log.warning('Failed to
[jira] [Comment Edited] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins edited comment on AIRFLOW-6515 at 4/1/20, 1:26 AM:
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, correct?
was (Author: wkhudgins):
[~toopt4] Sounds like this is essentially going through those and similar files
and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
>
[jira] [Comment Edited] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins edited comment on AIRFLOW-6515 at 4/1/20, 1:26 AM:
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, and if it doesn't, fixing the
log level, yes?
was (Author: wkhudgins):
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base,
[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins commented on AIRFLOW-6515:
---
[~toopt4] Sounds like this is essentially going through those and similar files
and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("""Missing
> attribute "%s" when looked-up in Ldap database.
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Parsing error
> when retrieving the user's group(s)."
>
[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072289#comment-17072289 ] Will Hudgins commented on AIRFLOW-6281: --- [~kamil.bregula] This is for the GCSToGCSOperator in [https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/gcs_to_gcs.py] ? If so I'd be willing to take it on. > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6295) Create guide for BigQuery service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072288#comment-17072288 ] Will Hudgins commented on AIRFLOW-6295: --- Beat me to it [~alexaabbas]!!! > Create guide for BigQuery service operator > -- > > Key: AIRFLOW-6295 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6295 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Assignee: Alexandra Abbas >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use BigQuery service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879789#comment-16879789 ] Will Hudgins commented on AIRFLOW-4816: --- [~jackjack10] Great, I'll move forward on it then: I just wanted to check since I am new to contributing. Thanks! > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Assignee: Will Hudgins >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879514#comment-16879514 ] Will Hudgins commented on AIRFLOW-4816: --- It seems like there is already a MySQLtoS3 operator in an Airflow plugin: [https://github.com/airflow-plugins/mysql_plugin/blob/master/operators/mysql_to_s3_operator.py] Do we really want/need to pull this into base Airflow? > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Assignee: Will Hudgins >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879415#comment-16879415 ] Will Hudgins commented on AIRFLOW-4816: --- [~lovk] What schema types would be desirable for export? CSV? JSON? XML? AVRO? > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Assignee: Will Hudgins >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879415#comment-16879415 ] Will Hudgins edited comment on AIRFLOW-4816 at 7/5/19 4:41 PM: --- [~lovk] What schema types would be desirable for export? CSV? JSON? XML? AVRO? I'm thinking for starters just JSON was (Author: wkhudgins): [~lovk] What schema types would be desirable for export? CSV? JSON? XML? AVRO? > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Assignee: Will Hudgins >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Hudgins reassigned AIRFLOW-4816: - Assignee: Will Hudgins > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Assignee: Will Hudgins >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4816) add MySQLToS3Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879365#comment-16879365 ] Will Hudgins commented on AIRFLOW-4816: --- I'll gladly take this issue on. > add MySQLToS3Operator > - > > Key: AIRFLOW-4816 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4816 > Project: Apache Airflow > Issue Type: Wish > Components: aws, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Priority: Major > > Airflow doesn't have MySQLToS3Operator > would be nice if someone can add it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
